ipython hangs on Auton cluster

[Chirag Nagpal]

Hi Viraj

The easiest option is to install anaconda on your home mounted on the NFS,
instead of separately installing conda for each cluster. (since each
cluster node is x64 you can expect conda compiled on one of the machines to
run on the other machines too. all the numpy math operations are taken care
of MKL/BLAS and so as long is MKL and BLAS in each machine are correctly
configured you will not experience a drop in performance)

Steps:

$wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
and then run
$bash https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh

Make sure after the installation is complete you export the environment
variable that points the PYTHONPATH to the conda directory. (The installer
will do this automatically in case it doesnt work you can export it in your
bashrc)

Chirag


On Tue, Aug 18, 2020 at 10:20 PM Viraj Mehta <virajm at andrew.cmu.edu> wrote:

> Hi Chirag,
>
> Where do you install your own Conda environment? Scratch? Any other tips
> on getting that done?
>
> Thanks,
> Viraj
>
> On Aug 18, 2020, at 9:19 PM, Chirag Nagpal <chiragn at cs.cmu.edu> wrote:
>
> FWIW  my recommendation is to set up your own conda environment and use
> the ipython version distributed with it. this way you can
> easily upgrade/modify your own python version without having to depend on
> the clusterwide distro
>
> On Tue, Aug 18, 2020 at 7:23 PM Predrag Punosevac <predragp at andrew.cmu.edu>
> wrote:
>
>> I looked a bit more carefully. It could be an upstream bug. It wouldn't
>> be the first time
>>
>> https://github.com/ipython/ipython/issues/11678
>>
>> You don't need ipython to run Python code. You could work and debug your
>> code on your local machine and just run production code on the server. A
>> typical python code is just a script starting with a shebang following with
>> a path to the binaries. I fail to see how ipython could be useful for that.
>> It is surely useful for the interactive work.
>>
>> Predrag
>>
>> On Tue, Aug 18, 2020 at 5:45 PM Viraj Mehta <virajm at andrew.cmu.edu>
>> wrote:
>>
>>> Tried this with 3.7 and 3.8 and it still hangs. Also if it’s a good
>>> clue, it doesn’t stop even if I send SIGINT or SIGQUIT. Not really sure
>>> what’s going on here.
>>>
>>> On Aug 18, 2020, at 4:39 PM, Viraj Mehta <virajm at andrew.cmu.edu> wrote:
>>>
>>> Yeah, I’ll give it a shot. Thanks!
>>>
>>> On Aug 18, 2020, at 4:38 PM, Predrag Punosevac <predragp at andrew.cmu.edu>
>>> wrote:
>>>
>>> I just upgraded all /opt/conda-py37 and /opt/conda-py38 packages on both
>>> GPU9 and GPU11. Could you please try again? Could you also try with py38
>>> which is now recommended and report back. If this works I will upgrade
>>> packages across all servers. This could be potentially remotely related to
>>> the fact that Ifegenia could not build TensorFlow. Another thought is that
>>> the ipython SQLite database is corrupted.
>>>
>>> Best,
>>> Predag
>>>
>>> On Tue, Aug 18, 2020 at 4:34 PM Viraj Mehta <virajm at andrew.cmu.edu>
>>> wrote:
>>>
>>>> Hi Predrag,
>>>>
>>>> Hope you’re doing well. I’ve been running into an issue the last couple
>>>> days on the Auton cluster that is blocking my work on code that used to
>>>> work and was hoping to get your thoughts. I have tried to distill this down
>>>> to a small but replicable issue, as seen in the attachment, which I have
>>>> seen hang on the ipython call on GPU9 and GPU11 so far. Do you know why
>>>> this might be? Thanks.
>>>>
>>>> Best,
>>>> Viraj
>>>
>>>
>>>
>>>
>
> --
>
> *Chirag Nagpal* PhD Student, Auton Lab
> School of Computer Science
> Carnegie Mellon University
> cs.cmu.edu/~chiragn
>
>
>

-- 

*Chirag Nagpal* PhD Student, Auton Lab
School of Computer Science
Carnegie Mellon University
cs.cmu.edu/~chiragn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200818/175f7892/attachment-0001.html>