Snippet: Size of PostgreSQL Tables using SqlAlchemy

This does a decent job of getting approximate sizes of tables and indexes in a PG table, using SA.

# SELECT pg_database_size('geekdb');
# with indexes
# SELECT pg_size_pretty(pg_total_relation_size('big_table'));
# without
# SELECT pg_size_pretty(pg_relation_size('big_table'));

from sqlalchemy import MetaData

dbparts = dict(user = some_user,
    pw = some_user,
    db = somedb,
    host = localhost)
dburi = "postgresql://%(user)s:%(pw)s@%(host)s/%(dd)s" % dbparts

def table_sizes(dburi,pretty=False):
    """ 
    Args:
        dburl.  string.  url of of the db to be reflected
        prettyprint. bool.  make the indexes into GB or whatnot
    
    Returns:
        dict::

            {table_name:  {'_total': int, '_table': int,
                'indexes':  {'idx1': int}}}
        
    Note:  Only works on PG, using PG specific items
    """
    out = dict()
    m = MetaData(dburi)
    e = lambda x: m.bind.execute(x).first()[0]
    m.reflect()
    q_pretty_total = "SELECT pg_size_pretty(pg_total_relation_size('%s'))"
    q_pretty_relation = "SELECT pg_size_pretty(pg_relation_size('%s'))"
    q_total = "SELECT pg_total_relation_size('%s')"
    q_relation = "SELECT pg_relation_size('%s')"

    for table_name,table in m.tables.iteritems():
        t = dict()
        if pretty:
            t['_total'] = e(q_pretty_total % table_name)
            t['_table'] = e(q_pretty_relation % table_name)
        else:
            t['_total'] = e(q_total % table_name)
            t['_table'] = e(q_relation % table_name)
        for idx in table.indexes:
            name = idx.name
            if pretty:
                t[name] = e(q_pretty_relation % name)
            else:
                t[name] = e(q_relation % name)
        
        out[table_name] = t
    
    return out


Tidbit: git-add –patch on a new file.

Today, I wanted to add a file to the git index interactively.

No problem!

$ echo blah > somefile
$ git-add --patch somefile
fatal: exec add--interactive failed.

It turns out there is a problem. Until git knows about the file, it can’t actually diff against it, or go into interactive patch mode. As it turns out, someone has thought ahead.

$ git add -N somefile  # adds it!

“-N” is the “–intent-to-add” flag. Whether this is a win or not is left as an exercise for the reader, but it does exist.


Bitten By SqlAlchemy 0.6 Truthiness

After upgrading to SqlAlchemy 0.6, I was bitten by this
known gotcha.

Given that st is some SqlAlchemy Table (as defined below):

import logging
from sqlalchemy import *
summary = Table(
    'summary',metadata,
    Column('country',String)
)
st = summary_table.c
query = select(st).where(st.country=='US')

Then we try to log using it

logging.critical('%r', query)

we get this uninformative error:

TypeError: Boolean value of this clause is not defined

What is going on here? It seems that as part of logging, the logger
must evaluate the truthiness of each argument to the logging call, as shown
in this part of the (wrapped) traceback:

/usr/lib64/python2.4/logging/__init__.pyc in __init__
    (self, name, level, pathname, lineno, msg, args, exc_info)
    224         # For the use case of passing a dictionary, this should not be a
    225         # problem.
--> 226         if args and (len(args) == 1) and args[0] and (type(args[0]) == types.DictType):
    227             args = args[0]
    228         self.args = args

As the Gotcha point out, while the stringiness of a Query is unchanged,
the truthiness is now in question. Luckily, the fix is simple (if hackish):

logging.critical('%s', str(query))

Recipe: All Linode StackScripts

Lately I’ve been playing around with Linode. One of the neat (but incomplete, hard to use, and not as good as it could be) features are Linode Stack Scripts. I wanted a full set of them to work with in constructing my own. Browsing through them graphically is a hassle, making it hard to grep them, copy from them (since they have line numbers, and the way to remove the line numbers is javascript), etc. This Python script downloads all of them to the cwd.

Season to taste, public domain license!

import re
import urllib
import htmlentitydefs

exp = re.compile(
   '''\[CDATA\[\n\n'''  + \
   '(.*?)' + \
   '\n\]\]>\\</div>',
   re.DOTALL | re.VERBOSE)


myrx = re.compile('&(' + '|'.join(htmlentitydefs.entitydefs.keys()) + ');')
def dehtml(s):
    return re.sub(
    myrx,
    lambda m: htmlentitydefs.entitydefs[m.group(1)],
    s
    )

last_seen = 0
ii = 0
while 1:
    ii += 1
    if ii - last_seen > 100:  
        print ("stopping at %i" % ii)
        break # we've probably seen them all
    try:
        text = urllib.urlopen("http://www.linode.com/stackscripts/view/?StackScriptID=%i#viewSource" % ii).read()
    except:
        continue
    
    try:
        ans = exp.search(text).groups()[0][1:]
    except:
        continue 
    
    ans = dehtml(ans)
    last_seen = ii
    print ("creating stackscript_%i.sh" % ii)
    ofh = file("stackscript_%i.sh" % ii,'w')
    ofh.write(ans)
    ofh.close()


Working Around WordPress

It’s official… I *hate* WordPress syntax and its blog entry box. As the years have gone on, I’ve gotten more annoyed with Wiki syntaxes in general. They all feel both overcomplicated and underpowered at the same time. In particular, both the Mediawiki and WordPress handling of *code* is horrible. I do like both POD and Restructured Text, and tend to write all my docs as .rst’s these days.

But Restructured Text has its own problems. Until there is a simple rst->html script (which I should get around to finding or writing), it’s a bit of a hassle to use.

So my current solution… use StackOverflow… that’s right, SO has an easy-to-amend, easy to use ReST-like syntax…. just use the “Ask A Question” box, hightlight the produced text (and in FF at least!) “View selection source” and paste it back in to the HTML entry area. Not trivial, but easy enough.


An IanB-tastic day, or How A Bug Becomes a Fix

People don’t write enough about how they catch, report, and fix bugs. I hope others will follow my lead in exposing the process more.

  1. Tried to read IanB’s revised webob tutorial at
    http://pythonpaste.org/webob/do-it-yourself.html
  2. where I got annoyed by how copying and pasting the code is hard
    with the “>>>” and “…” symbols. At the first example!
  3. Since Python-Sphinx is the issue, spent some time in #python-docs
    discussing soluions with Taggnostr, including ones that other
    code highlighters use.
  4. Thought about how IanB probably likes that the code isn’t just
    cut and pastable, since it typing it in yourself is much better for learning.
    Decided that I didn’t care!
  5. Built on Taggnostr’s jquery-based fix on the installed
    doctools.js file.
  6. Branched and checked out Sphinx from BitBucket to work on it
    it more formally…
  7. where I promptly made a mess of things. I don’t know jquery or
    javascript very well, so there was a lot of fussing. The problem
    was that between my version and tip, underscore.js was added, so
    using the new doctools.js file in my generated sphinx html tree
    was causing some silent errors! Boy, JS seems to be hard to troubleshoot,
    and not very good at failing loudly!
  8. After finishing my fixes, checked in the fix to BitBucked and pushed.
  9. Made a pull-request, where I discovered that then you pull on your own
    branch at BB, it sends the request to you. This seems, hm…, unintuitive!
  10. After asking about it at #mercurial who answered, and #bitbucket who didn’t,
    people agreed that this was, um, odd behaviour.
  11. So, bug time at BitBucket, where after a search, I found Bug 681…
    (http://bitbucket.org/jespern/bitbucket/issue/681/master-repositories-dont-need-pull-request)
  12. …which was filed by IanB!

Whython – Python For People Who Hate Whitespace

Whython : Whitespace Haters Python

http://writeonly.files.wordpress.com/2010/04/whython.png?w=590

Example

Clearly Confusing (standard 3.x):

for ii in range(10):
    print(ii)
    print("which is %s" % (['even','odd'][ii % 2]))

Improved:

for ii in range(10) {
    print(ii);
    print("which is %s" % (['even','odd'][ii % 2]));
}

Maximum Enterprise Whythonic:

for ii in range(10) { print(ii); print("which is %s" % (['even','odd'][ii % 2])); }

How about some Scheme with your Python?

defun myfun():  return 1
assert myfun() == 1

Or add some Ruby shine?

def myfun() BEGIN return 1; END
assert myfun() == 1

Why Whython?

  • Less Whitespace, More Enterprise
  • It’s not a real language without braces and semi colons
  • Whitespace delimited is like so restrictive, man!
  • Python sucks for code golf
  • Finally, a Python for everyone who can’t decide between tab and space
  • Possibly (as in the mathematical sense – a small non-zero probability)
    useful for doing command line one liners in python
  • Help determine how bad a PEP/developement idea needs to be before
    someone gets kickbanned from #python-dev.

More seriously

  • reading the Dragon Book [Aho86] gives a person dangerous ideas
  • good excuse to deep dive into the python interpreter source code and the AST, dis modules
  • finally wanted to learn GDB and python -d debug mode
  • humoring trolls is fun
  • for education, the whitespace thing really can cause problems. When
    copying code out of books into IDLE or IPython, there are corner cases when
    it terminates blocks “too early”, confusing new learners.
  • preparation for the “Python Spring Cleaning” project, to see how hard it is
    to get and modify source, write a PEP, raise bug ideas, talk in irc, etc.
  • since this is unlikely to ever be adopted by Python (I hope!), it will
    remain a useful exercise, unlike othe “bugs” which get fixed once and for
    all

Want It? (Download and Install)

Are you sure you can handle this level of awesome? Okay! Download and install:

http://bitbucket.org/gregglind/python-whython3k/src/

## Get the source!
$ hg clone https://gregglind@bitbucket.org/gregglind/python-whython3k/
    # or if you haven't jumped on the `Mercurial <http://mercurial.selenic.com/wiki/Tutorial>`_  bandwagon
    # then:  wget http://bitbucket.org/gregglind/python-whython3k/get/79a2c77fe3e1.zip and unzip it!
$ cd python-whython3k
$ configure  # go make a pot of tea
$ make       # go watch an episode of the `IT Crowd <http://www.netflix.com/WiMovie/The_IT_Crowd_Series_1/70113774>`_
$ ./whython  # beautiful failure begins

Limitations

  • only simple_stmt are really usable in this way. That means that
    blocks (functions, if, else, etc.) can’t be nested inside a braced block.

Thanks to

  • The Authors of PEP 306
  • GVR, Martin v. Loewis (my umlaut is misbehaving!), Georg Brandl, Greg Ewing, Jeremy Hylton and others on the
    Python-Dev mailing list
  • Fred Drake, for responding to my crazy and incoherent email
  • gutworth, merwok, __ap__ and others in #python-dev

References

[Aho86] Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman.
Compilers: Principles, Techniques, and Tools,
http://www.amazon.com/exec/obidos/tg/detail/-/0201100886/104-0162389-6419108

Follow

Get every new post delivered to your Inbox.