<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi All,<div class=""><br class=""></div><div class="">For your information, for those who are using jupyter notebook, you might have to do the following too:</div><div class=""><br class=""></div><div class="">1. <span style="color: rgb(51, 51, 51); font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(248, 248, 248);" class="">jupyter notebook --generate-config</span></div><div class=""><div>2. you will find jupyter notebook config at <span style="color: rgb(61, 60, 64); font-family: "Open Sans", sans-serif; font-size: 13.5px; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(255, 255, 255);" class="">~/.jupyter/jupyter_notebook_config.py</span></div><div><span style="color: rgb(61, 60, 64); font-family: "Open Sans", sans-serif; font-size: 13.5px; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(255, 255, 255);" class="">3. set </span><span style="color: rgb(51, 51, 51); font-family: Menlo, Monaco, Consolas, "Courier New", monospace; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(248, 248, 248);" class="">c.NotebookNotary.db_file = ':memory:'</span></div><div><br class=""></div><div>I also updated my nbformat to the latest version. Issue related here: <a href="https://github.com/jupyter/nbformat/issues/52" class="">https://github.com/jupyter/nbformat/issues/52</a></div><div><br class=""></div><div>Cheers,</div><div>Sitong</div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On 20 Aug 2020, at 07:35, Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" class="">virajm@andrew.cmu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Actually, I figured this out: for everyone having trouble with iPython/jupyter, here’s a solution:</span><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=""></div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">1. Get into a python environment that has ipython installed</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">2. Run `ipython profile create`</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">3. Run `cd ~/.ipython/profile_default`</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">4. Edit the file in there called ipython_config.py by finding the option for c.HistoryAccessor.hist_file and setting it to ‘:memory:’ ( this will mean your command history isn’t saved between ipython sessions but whatever, you could also point this at scratch).</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=""></div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Hope this is helpful — not sure how to add this to the wiki but it might be good to do so.</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=""></div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Viraj<br class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 19, 2020, at 5:46 PM, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" class="">predragp@andrew.cmu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class="">Your report indicates that my gut feeling that SQLite database is the culprit seems to be correct. Per our documentation<div class=""><br class=""></div><div class=""><a href="https://www.autonlab.org/autonlab_wiki/aetiquette.html#don-ts" class="">https://www.autonlab.org/autonlab_wiki/aetiquette.html#don-ts</a><br class=""></div><div class=""><br class=""></div><div class=""><span class="" style="font-size: 16px; color: rgb(64, 64, 64); font-family: Lato, proxima-nova, "Helvetica Neue", Arial, sans-serif; background-color: rgb(252, 252, 252);"><b class="">Use your scratch directory to store Jupiter sqlite database!</b></span><br class=""></div><div class=""><span class="" style="font-size: 16px; color: rgb(64, 64, 64); font-family: Lato, proxima-nova, "Helvetica Neue", Arial, sans-serif; background-color: rgb(252, 252, 252);"><b class=""><br class=""></b></span></div><div class=""><font color="#404040" face="Lato, proxima-nova, Helvetica Neue, Arial, sans-serif" class=""><span class="" style="background-color: rgb(252, 252, 252);">You placed your SQLite database onto the NFS share (zfsauton2) and you are surprised that it is incoherent. I hope you understand now better the lack of urgency in my responses. </span></font></div><div class=""><font color="#404040" face="Lato, proxima-nova, Helvetica Neue, Arial, sans-serif" class=""><span class="" style="background-color: rgb(252, 252, 252);"><br class=""></span></font></div><div class=""><font color="#404040" face="Lato, proxima-nova, Helvetica Neue, Arial, sans-serif" class=""><span class="" style="background-color: rgb(252, 252, 252);">Predrag</span></font></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 19, 2020 at 6:03 PM Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" class="">virajm@andrew.cmu.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;">Hi Predrag & Users,<div class=""><br class=""></div><div class="">I have a clue as to what is wrong with our cluster. Had a few processes running which broke due to this sqlite error from ipython: <span id="cid:17408e010ab5b206ef61" class=""><PastedGraphic-1.png></span></div><div class="">I’d imagine this is what is wrong with all our ipython stuff. No idea how to debug this, but I hope it can be helpful as we try to fix this.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Viraj</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 18, 2020, at 10:28 PM, Chufan Gao <<a href="mailto:chufang@andrew.cmu.edu" target="_blank" class="">chufang@andrew.cmu.edu</a>> wrote:</div><br class=""><div class=""><div id="gmail-m_-5321130582129444634divtagdefaultwrapper" dir="ltr" class="" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div class="" style="margin-top: 0px; margin-bottom: 0px;">Hi All,</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div dir="ltr" class=""><div id="gmail-m_-5321130582129444634x_divtagdefaultwrapper" dir="ltr" class="" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div class="" style="margin-top: 0px; margin-bottom: 0px;">Rachel and I are also facing a similar issue with our Jupyter notebooks. </div><div class="" style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">We also both reinstalled jupyter with no effect.</div><p class="" style="margin-top: 0px; margin-bottom: 0px;"></p><div class="" style="margin-top: 0px; margin-bottom: 0px;">For me, these notebooks are extremely helpful in fast code iteration and testing out concepts.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;">I also have the intuition that it is an upstream issue, as they were running fine (without any changes) before lop2 went down.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Would you please take another look?</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Worst case, I have to convert my notebooks into .py files, which will slow things down.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div id="gmail-m_-5321130582129444634x_Signature" class=""><div id="gmail-m_-5321130582129444634x_divtagdefaultwrapper" dir="ltr" class="" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div class="" style="margin-top: 0px; margin-bottom: 0px;">Sincerely,</div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Chufan (Andy) Gao<br class=""></div></div></div></div><hr class="" style="display: inline-block; width: 853.578125px;"><div id="gmail-m_-5321130582129444634x_divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" class="" style="font-size: 11pt;"><b class="">From:</b><span class=""> </span>Autonlab-users <<a href="mailto:autonlab-users-bounces@autonlab.org" target="_blank" class="">autonlab-users-bounces@autonlab.org</a>> on behalf of Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">predragp@andrew.cmu.edu</a>><br class=""><b class="">Sent:</b><span class=""> </span>Tuesday, August 18, 2020 10:35:11 PM<br class=""><b class="">To:</b><span class=""> </span>Viraj Mehta<br class=""><b class="">Cc:</b><span class=""> </span><a href="mailto:users@autonlab.org" target="_blank" class="">users@autonlab.org</a><br class=""><b class="">Subject:</b><span class=""> </span>Re: ipython hangs on Auton cluster</font><div class=""> </div></div></div><font size="2" class=""><span class="" style="font-size: 10pt;"><div class="">Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">virajm@andrew.cmu.edu</a>> wrote:<br class=""><br class="">> I???m pretty sure it???s not an upstream bug, as many environments<br class="">> (conda and virtualenv) which were working with ipython across several<br class="">> python versions before are now not working.<br class="">><span class=""> </span><br class="">> I understand that ipython and ipdb aren???t typically required for<br class="">> Python workflows but certain efforts, like stepping through code that<br class="">> requires a GPU and loads a model from the Auton cluster, are difficult<br class="">> to debug without ipdb. Is there anything else that has changed that<br class="">> might have broken it?<br class=""><br class="">Nothing that I am aware of. However, you do understand that the system<br class="">is very complex and it is like a live organism constantly morphing.<br class=""><br class="">Best,<br class="">Predrag<br class=""><br class=""><br class=""><br class="">><span class=""> </span><br class="">> Thanks,<br class="">> Viraj<br class="">><span class=""> </span><br class="">> > On Aug 18, 2020, at 6:21 PM, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">predragp@andrew.cmu.edu</a>> wrote:<br class="">> ><span class=""> </span><br class="">> > I looked a bit more carefully. It could be an upstream bug. It wouldn't be the first time<br class="">> ><span class=""> </span><br class="">> ><span class=""> </span><a href="https://github.com/ipython/ipython/issues/11678" id="gmail-m_-5321130582129444634LPlnk209515" target="_blank" class="">https://github.com/ipython/ipython/issues/11678</a><span class=""> </span><<a href="https://github.com/ipython/ipython/issues/11678" id="gmail-m_-5321130582129444634LPlnk702306" target="_blank" class="">https://github.com/ipython/ipython/issues/11678</a>><div id="gmail-m_-5321130582129444634LPBorder_GT_15978075416460.6993493734110094" class="" style="margin-bottom: 20px; overflow: auto; width: 871px; text-indent: 0px;"><table id="gmail-m_-5321130582129444634LPContainer_15978075416430.40678795686000635" cellspacing="0" class="" style="width: 783.890625px; background-color: rgb(255, 255, 255); overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top-width: 1px; border-top-style: dotted; border-top-color: rgb(200, 200, 200); border-bottom-width: 1px; border-bottom-style: dotted; border-bottom-color: rgb(200, 200, 200);"><tbody class=""><tr valign="top" class="" style="border-spacing: 0px;"><td id="gmail-m_-5321130582129444634ImageCell_15978075416430.03874342033191813" colspan="1" class="" style="width: 250px; display: table-cell; padding-right: 20px;"><div id="gmail-m_-5321130582129444634LPImageContainer_15978075416430.3005447598277968" class="" style="background-color: rgb(255, 255, 255); height: 250px; margin: auto; display: table; width: 250px;"><a id="gmail-m_-5321130582129444634LPImageAnchor_15978075416440.2677800662759784" href="https://github.com/ipython/ipython/issues/11678" target="_blank" class="" style="display: table-cell; text-align: center;"><img id="gmail-m_-5321130582129444634LPThumbnailImageID_15978075416440.6841686551198707" width="250" height="250" src="https://avatars1.githubusercontent.com/u/230453?s=400&v=4" class="" style="display: inline-block; max-width: 250px; max-height: 250px; height: 250px; width: 250px; border-width: 0px; vertical-align: bottom;"></a></div></td><td id="gmail-m_-5321130582129444634TextCell_15978075416440.5034809612202791" colspan="2" class="" style="vertical-align: top; padding: 0px; display: table-cell;"><div id="gmail-m_-5321130582129444634LPRemovePreviewContainer_15978075416440.7309585310895517" class=""></div><div id="gmail-m_-5321130582129444634LPTitle_15978075416440.29044991062034464" class="" style="color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; line-height: 21px;"><a id="gmail-m_-5321130582129444634LPUrlAnchor_15978075416450.9721070572969626" href="https://github.com/ipython/ipython/issues/11678" target="_blank" class="" style="text-decoration: none;">ipython won't start · Issue #11678 · ipython/ipython · GitHub</a></div><div id="gmail-m_-5321130582129444634LPMetadata_15978075416450.1478588093826021" class="" style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 14px;"><a href="http://github.com/" target="_blank" class="">github.com</a></div><div id="gmail-m_-5321130582129444634LPDescription_15978075416450.16188406407175893" class="" style="display: block; color: rgb(102, 102, 102); font-weight: 400; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 20px; max-height: 100px; overflow: hidden;">Now I'm facing that ipython won't start without any error messages. I tried to run it with DEBUG, then the command will be "uninterruptible sleep" after the logs. $ pyenv global system $ python --version Python 2.7.5 $ ipython --version ...</div></td></tr></tbody></table></div><br class=""><div id="gmail-m_-5321130582129444634LPBorder_GT_15978075416030.23597890184593506" class="" style="margin-bottom: 20px; overflow: auto; width: 871px; text-indent: 0px;"><table id="gmail-m_-5321130582129444634LPContainer_15978075415930.2997923841902339" cellspacing="0" class="" style="width: 783.890625px; background-color: rgb(255, 255, 255); overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top-width: 1px; border-top-style: dotted; border-top-color: rgb(200, 200, 200); border-bottom-width: 1px; border-bottom-style: dotted; border-bottom-color: rgb(200, 200, 200);"><tbody class=""><tr valign="top" class="" style="border-spacing: 0px;"><td id="gmail-m_-5321130582129444634ImageCell_15978075415960.18997476242517464" colspan="1" class="" style="width: 250px; display: table-cell; padding-right: 20px;"><div id="gmail-m_-5321130582129444634LPImageContainer_15978075415960.515460133509023" class="" style="background-color: rgb(255, 255, 255); height: 250px; margin: auto; display: table; width: 250px;"><a id="gmail-m_-5321130582129444634LPImageAnchor_15978075415970.3517790927316342" href="https://github.com/ipython/ipython/issues/11678" target="_blank" class="" style="display: table-cell; text-align: center;"><img id="gmail-m_-5321130582129444634LPThumbnailImageID_15978075415970.5405652912323857" width="250" height="250" src="https://avatars1.githubusercontent.com/u/230453?s=400&v=4" class="" style="display: inline-block; max-width: 250px; max-height: 250px; height: 250px; width: 250px; border-width: 0px; vertical-align: bottom;"></a></div></td><td id="gmail-m_-5321130582129444634TextCell_15978075415980.6552365384153627" colspan="2" class="" style="vertical-align: top; padding: 0px; display: table-cell;"><div id="gmail-m_-5321130582129444634LPRemovePreviewContainer_15978075415990.01438989417682146" class=""></div><div id="gmail-m_-5321130582129444634LPTitle_15978075415990.9198803079481062" class="" style="color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; line-height: 21px;"><a id="gmail-m_-5321130582129444634LPUrlAnchor_15978075416000.5508698699267309" href="https://github.com/ipython/ipython/issues/11678" target="_blank" class="" style="text-decoration: none;">ipython won't start · Issue #11678 · ipython/ipython · GitHub</a></div><div id="gmail-m_-5321130582129444634LPMetadata_15978075416000.3645122326708419" class="" style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 14px;"><a href="http://github.com/" target="_blank" class="">github.com</a></div><div id="gmail-m_-5321130582129444634LPDescription_15978075416010.8819176189432545" class="" style="display: block; color: rgb(102, 102, 102); font-weight: 400; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 20px; max-height: 100px; overflow: hidden;">Now I'm facing that ipython won't start without any error messages. I tried to run it with DEBUG, then the command will be "uninterruptible sleep" after the logs. $ pyenv global system $ python --version Python 2.7.5 $ ipython --version ...</div></td></tr></tbody></table></div><br class=""><br class="">> ><span class=""> </span><br class="">> > You don't need ipython to run Python code. You could work and debug your code on your local machine and just run production code on the server. A typical python code is just a script starting with a shebang following with a path to the binaries. I fail to see how ipython could be useful for that. It is surely useful for the interactive work.<span class=""> </span><br class="">> ><span class=""> </span><br class="">> > Predrag<br class="">> ><span class=""> </span><br class="">> > On Tue, Aug 18, 2020 at 5:45 PM Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">virajm@andrew.cmu.edu</a><span class=""> </span><<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">mailto:virajm@andrew.cmu.edu</a>>> wrote:<br class="">> > Tried this with 3.7 and 3.8 and it still hangs. Also if it???s a good clue, it doesn???t stop even if I send SIGINT or SIGQUIT. Not really sure what???s going on here.<br class="">> ><span class=""> </span><br class="">> >> On Aug 18, 2020, at 4:39 PM, Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">virajm@andrew.cmu.edu</a><span class=""> </span><<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">mailto:virajm@andrew.cmu.edu</a>>> wrote:<br class="">> >><span class=""> </span><br class="">> >> Yeah, I???ll give it a shot. Thanks!<br class="">> >><span class=""> </span><br class="">> >>> On Aug 18, 2020, at 4:38 PM, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">predragp@andrew.cmu.edu</a><span class=""> </span><<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">mailto:predragp@andrew.cmu.edu</a>>> wrote:<br class="">> >>><span class=""> </span><br class="">> >>> I just upgraded all /opt/conda-py37 and /opt/conda-py38 packages on both GPU9 and GPU11. Could you please try again? Could you also try with py38 which is now recommended and report back. If this works I will upgrade packages across all servers. This could be potentially remotely related to the fact that Ifegenia could not build TensorFlow. Another thought is that the ipython SQLite database is corrupted.<span class=""> </span><br class="">> >>><span class=""> </span><br class="">> >>> Best,<br class="">> >>> Predag<br class="">> >>><span class=""> </span><br class="">> >>> On Tue, Aug 18, 2020 at 4:34 PM Viraj Mehta <<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">virajm@andrew.cmu.edu</a><span class=""> </span><<a href="mailto:virajm@andrew.cmu.edu" target="_blank" class="">mailto:virajm@andrew.cmu.edu</a>>> wrote:<br class="">> >>> Hi Predrag,<br class="">> >>><span class=""> </span><br class="">> >>> Hope you???re doing well. I???ve been running into an issue the last couple days on the Auton cluster that is blocking my work on code that used to work and was hoping to get your thoughts. I have tried to distill this down to a small but replicable issue, as seen in the attachment, which I have seen hang on the ipython call on GPU9 and GPU11 so far. Do you know why this might be? Thanks.<br class="">> >>><span class=""> </span><br class="">> >>> Best,<br class="">> >>> Viraj<br class="">> >><span class=""> </span><br class="">> ><span class=""> </span><br class="">></div></span></font></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></div></body></html>