ipython hangs on Auton cluster

[Predrag Punosevac]

Viraj Mehta <virajm at andrew.cmu.edu> wrote:

> I???m pretty sure it???s not an upstream bug, as many environments
> (conda and virtualenv) which were working with ipython across several
> python versions before are now not working.
> 
> I understand that ipython and ipdb aren???t typically required for
> Python workflows but certain efforts, like stepping through code that
> requires a GPU and loads a model from the Auton cluster, are difficult
> to debug without ipdb. Is there anything else that has changed that
> might have broken it?

Nothing that I am aware of. However, you do understand that the system
is very complex and it is like a live organism constantly morphing.

Best,
Predrag



> 
> Thanks,
> Viraj
> 
> > On Aug 18, 2020, at 6:21 PM, Predrag Punosevac <predragp at andrew.cmu.edu> wrote:
> > 
> > I looked a bit more carefully. It could be an upstream bug. It wouldn't be the first time
> > 
> > https://github.com/ipython/ipython/issues/11678 <https://github.com/ipython/ipython/issues/11678>
> > 
> > You don't need ipython to run Python code. You could work and debug your code on your local machine and just run production code on the server. A typical python code is just a script starting with a shebang following with a path to the binaries. I fail to see how ipython could be useful for that. It is surely useful for the interactive work. 
> > 
> > Predrag
> > 
> > On Tue, Aug 18, 2020 at 5:45 PM Viraj Mehta <virajm at andrew.cmu.edu <mailto:virajm at andrew.cmu.edu>> wrote:
> > Tried this with 3.7 and 3.8 and it still hangs. Also if it???s a good clue, it doesn???t stop even if I send SIGINT or SIGQUIT. Not really sure what???s going on here.
> > 
> >> On Aug 18, 2020, at 4:39 PM, Viraj Mehta <virajm at andrew.cmu.edu <mailto:virajm at andrew.cmu.edu>> wrote:
> >> 
> >> Yeah, I???ll give it a shot. Thanks!
> >> 
> >>> On Aug 18, 2020, at 4:38 PM, Predrag Punosevac <predragp at andrew.cmu.edu <mailto:predragp at andrew.cmu.edu>> wrote:
> >>> 
> >>> I just upgraded all /opt/conda-py37 and /opt/conda-py38 packages on both GPU9 and GPU11. Could you please try again? Could you also try with py38 which is now recommended and report back. If this works I will upgrade packages across all servers. This could be potentially remotely related to the fact that Ifegenia could not build TensorFlow. Another thought is that the ipython SQLite database is corrupted. 
> >>> 
> >>> Best,
> >>> Predag
> >>> 
> >>> On Tue, Aug 18, 2020 at 4:34 PM Viraj Mehta <virajm at andrew.cmu.edu <mailto:virajm at andrew.cmu.edu>> wrote:
> >>> Hi Predrag,
> >>> 
> >>> Hope you???re doing well. I???ve been running into an issue the last couple days on the Auton cluster that is blocking my work on code that used to work and was hoping to get your thoughts. I have tried to distill this down to a small but replicable issue, as seen in the attachment, which I have seen hang on the ipython call on GPU9 and GPU11 so far. Do you know why this might be? Thanks.
> >>> 
> >>> Best,
> >>> Viraj
> >> 
> > 
>