Changeset 2286 for trunk/python/scantable.py

Timestamp:

09/02/11 19:05:11 (13 years ago)

Author:

Kana Sugimoto

Message:

New Development: No (performance tuning)

JIRA Issue: No

Ready for Test: Yes

Interface Changes: Yes

What Interface Changed: a parameter "filename" is added to Scantable::summary. scantable.summary doesn't return a string anymore

Test Programs: sdlist unittest/ scantable.summary("summary.txt")

Put in Release Notes: Yes

Module(s): sdlist, asap.summary

Description:

scantable.summary is very slow for large data sets (in row number) often outputted
by modern telescopes. It takes > 1.5 hours to list OTF raster scan with 350,000 rows.

This was because, the methods accumulates the whole text string (~700,000 lines) and
returns it as a string. Once the summary string exceed several tens thousands lines,
elapse time increases non-linearly, may be because very massive output string starts
to overweigh the memory.

I updated scantable.summary so that it flushes the summary string more often to file/logger.
After the modification, scantable.summary could list the data mentioned above in ~ 7 minutes.
The side effect of it is that scantable.summary doesn't return summary string anymore.
(But people may not happy with sub-million lines of string anyway.)

File:

• trunk/python/scantable.py (modified) (2 diffs)

trunk/python/scantable.py

@@ -445 +445 @@
 
         """
-        info = Scantable._summary(self)
+#         info = Scantable._summary(self)
         if filename is not None:
             if filename is "":
@@ -451 +451 @@
             from os.path import expandvars, isdir
             filename = expandvars(filename)
-            if not isdir(filename):
-                data = open(filename, 'w')
-                data.write(info)
-                data.close()
-            else:
+#             if not isdir(filename):
+#                 data = open(filename, 'w')
+#                 data.write(info)
+#                 data.close()
+#             else:
+            if isdir(filename):
                 msg = "Illegal file name '%s'." % (filename)
                 raise IOError(msg)
-        return page(info)
+        else:
+            filename = ""
+        Scantable._summary(self, filename)
+#         return page(info)
 
     def get_spectrum(self, rowno):