Monday, November 25, 2013

Trigonometry and Nested Radicals

Early last month, I was chatting with one of my officemates about a curious problem I had studied in high school. I hadn't written any of the results down, so much of the discussion involved me rediscovering the results and proving them with much more powerful tools than I once possessed.

Before writing about the problem I had played around with, I want to give a brief motivation. For as long as humans have been doing mathematics, finding values of \(\pi\) has been deemed worthwhile (or every generation has just found it worthwhile to waste time computing digits).

One such way the Greeks (particularly Archmides) computed \(\pi\) was by approximating a circle by a regular polygon and letting the number of sides grow large enough so that the error between the area of the unit circle (\(\pi \cdot 1^2\)) and the area of the polygon would be smaller than some fixed threshold. Usually these thresholds were picked to ensure that the first \(k\) digits were fully accurate (for some appropriate value of \(k\)).

In many introductory Calculus courses, this problem is introduced exactly when the limit is introduced and students are forced to think about the area problem in the regular polygon:
Given \(N\) sides, the area is \(N \cdot T_N\) where \(T_N\) is the area of each individual triangle given by one side of the polygon and the circumcenter.

Call one such triangle \(\Delta ABC\) and let \(BC\) be the side that is also a side of the polygon while the other sides have \(\left|AB\right| = \left|AC\right| = 1\) since the polygon is inscribed in a unit circle. The angle \(\angle BAC = \frac{2\pi}{N}\) since each of the triangles has the same internal angle and there are \(N\) of them. If we can find the perpendicular height \(h\) from \(AB\) to \(C\), the area will be \(\frac{1}{2} h \left|AB\right| = \frac{h}{2}\). But we also know that
\[\sin\left(\angle BAC\right) = \frac{h}{\left|AC\right|} \Rightarrow h = \sin\left(\frac{2\pi}{N}\right).\] Combining all of these, we can approximate \(\pi\) with the area:
\[\pi \approx \frac{N}{2} \sin\left(\frac{2\pi}{N}\right) = \pi \frac{\sin\left(\frac{2\pi}{N}\right)}{\frac{2 \pi}{N}}. \] As I've shown my Math 1A students, we see that
\[\lim_{N \to \infty} \pi \frac{\sin\left(\frac{2\pi}{N}\right)}{\frac{2 \pi}{N}} = \pi \lim_{x \to 0} \frac{\sin(x)}{x} = \pi\] so these are indeed good approximations.

Theory is Nice, But I Thought We Were Computing Something

Unfortunately for us (and Archimedes), computing \(\sin\left(\frac{2\pi}{N}\right)\) is not quite as simple as dividing by \(N\), so often special values of \(N\) were chosen. In fact, starting from \(N\) and then using \(2N\), the areas could be computed via a special way of averaging the previous areas. Lucky for us, such a method is equivalent to the trusty half angle identities (courtesy of Abraham De Moivre). To keep track of these polygons with a power of two as the number of sides, we call \(A_n = \frac{2^n}{2} \sin\left(\frac{2\pi}{2^n}\right)\).

Starting out with the simplest polygon, the square with \(N = 2^2\) sides, we have
\[A_2 = 2 \sin\left(\frac{\pi}{2}\right) = 2.\] Jumping to the octagon (no not that "The Octagon"), we have
\[A_3 = 4 \sin\left(\frac{\pi}{4}\right) = 4 \frac{\sqrt{2}}{2} = 2 \sqrt{2}.\] So far, the toughest thing we've had to deal with is a \(45^{\circ}\) angle and haven't yet had to lean on Abraham (himnot him) for help. The hexadecagon wants to change that:
\[A_4 = 8 \sin\left(\frac{\pi}{8}\right) = 8 \sqrt{\frac{1 - \cos\left(\frac{\pi}{4}\right)}{2}} = 8 \sqrt{\frac{2 - \sqrt{2}}{4}} = 4 \sqrt{2 - \sqrt{2}}.\]
To really drill home the point (and motivate my next post) we'll compute this for the \(32\)-gon (past the point where polygons have worthwhile names):
\[A_5 = 16 \sin\left(\frac{\pi}{16}\right) = 16 \sqrt{\frac{1 - \cos\left(\frac{\pi}{8}\right)}{2}}.\] Before, we could rely on the fact that we know that a \(45-45-90\) triangle looked like, but now, we come across \(\cos\left(\frac{\pi}{8}\right)\), a value which we haven't seen before. Luckily, Abraham has help here as well:
\[\cos\left(\frac{\pi}{8}\right) = \sqrt{\frac{1 + \cos\left(\frac{\pi}{4}\right)}{2}} = \sqrt{\frac{2 + \sqrt{2}}{4}} = \frac{1}{2} \sqrt{2 + \sqrt{2}}\] which lets us compute
\[A_5 = 16 \sqrt{\frac{1 - \frac{1}{2} \sqrt{2 + \sqrt{2}}}{2}} = 8 \sqrt{2 - \sqrt{2 + \sqrt{2}}}.\]

So why have I put you through all this? If we wave our hands like a magician, we can see this pattern continues and for the general \(n\)
\[A_n = 2^{n - 2} \sqrt{2 - \sqrt{2 + \sqrt{2 + \sqrt{\cdots + \sqrt{2}}}}}\]
where there are \(n - 3\) nested radicals with the \(\oplus\) sign and only one minus sign at the beginning.

This motivates us to study two questions, what is the limiting behavior of such a nested radical:
\[\sqrt{2 + s_1 \sqrt{2 + s_2 \sqrt{ \cdots }}}\] as the signs \(s_1, s_2, \ldots\) takes values in \(\left\{-1, 1\right\}\). Recasting in terms of the discussion above, we want to know how close we are to \(\pi\) as we increase the number of sides.

When I was in high school, I just loved to nerd out on any and all math problems, so I studied this just for fun. Having heard about the unfathomable brain of Ramanujan and the fun work he had done with infinitely nested radicals, I wanted to examine which sequences of signs \((s_1, s_2, \ldots)\) produced an infinite radical that converged and what the convergence behavior was.

I'm fairly certain my original questions came from an Illinois Council of Teachers of Mathematics (ICTM) contest problem along the lines of
\[\text{Find the value of the infinite nested radical } \sqrt{2 + \sqrt{2 + \cdots}}\] or maybe the slightly more difficult \[\text{Find the value of the infinite nested radical } \sqrt{2 - \sqrt{2 + \sqrt{2 - \sqrt{2 + \cdots}}}}.\] Armed with my TI-83, I set out to do some hardcore programming and figure it out. It took me around a month of off-and-on tinkering. This second time around as a mathematical grown-up, it took me the first half of a plane ride from SFO to Dallas.

In the next few weeks/months, I hope to write a few blog posts, including math, proofs and some real code on what answers I came up with and what other questions I have.

Tuesday, September 10, 2013

Calculating a Greatest Common Divisor with Dirichlet's Help

Having just left Google and started my PhD in Applied Mathematics at Berkeley, I thought it might be appropriate to write some (more) math-related blog posts. Many of these posts, I jotted down on napkins and various other places on the web and just haven't had time to post until now.

For today, I'm posting a result which was somewhat fun to figure out with/for one of my buddies from Michigan Math. I'd also like to point out that he is absolutely kicking ass at Brown.

While trying to determine if
\[J(B_n)_{\text{Tor}}\left(\mathbb{Q}\right) \stackrel{?}{=} \mathbb{Z}/2\mathbb{Z} \] where \(J(B_n)\) is the Jacobian of the curve \(B_n\) given by \(y^2 = (x + 2) \cdot f^n(x)\) where \(f^n\) denotes \(\underbrace{f \circ \cdots \circ f}_{n \text{ times}}\) and \(f(x) = x^2 - 2\).

Now, his and my interests diverged some time ago, so I can't appreciate what steps took him from this to the problem I got to help with. However, he was able to show (trivially maybe?) that this was equivalent to showing that
\[\gcd\left(5^{2^n} + 1, 13^{2^n} + 1, \ldots, p^{2^n} + 1, \ldots \right) = 2 \qquad (1)\] where the \(n\) in the exponents is the same as that in \(B_n\) and where the values we are using in our greatest common divisor (e.g. \(5, 13\) and \(p\) above) are all of the primes \(p \equiv 5 \bmod{8}\).

My buddy, being sadistic and for some reason angry with me, passed me along the stronger statement:
\[\gcd\left(5^{2^n} + 1, 13^{2^n} + 1\right) = 2 \qquad (2)\] which I of course struggled with and tried to beat down with tricks like \(5^2 + 12^2 = 13^2\). After a few days of this struggle, he confessed that he was trying to ruin my life and told me about the weaker version \((1)\).

When he sent me the email informing me of this, I read it at 8am, drove down to Santa Clara for PyCon and by the time I arrived at 8:45am I had figured the weaker case \((1)\) out. This felt much better than the days of struggle and made me want to write about my victory (which I'm doing now). Though, before we actually demonstrate the weaker fact \((1)\)  I will admit that I am not in fact tall. Instead I stood on the shoulders of Dirichlet and called myself tall. Everything else is bookkeeping.

Let's Start the Math

First, if \(n = 0\), we see trivially that
\[\gcd\left(5^{2^0} + 1, 13^{2^0} + 1\right) = \gcd\left(6, 14\right) = 2\] and all the remaining terms are divisible by \(2\) hence the \(\gcd\) over all the primes must be \(2\).

Now, if \(n > 0\), we will show that \(2\) divides our \(\gcd\), but \(4\) does not and that no odd prime can divide this \(\gcd\). First, for \(2\), note that
\[p^{2^n} + 1 \equiv \left(\pm 1\right)^{2^n} + 1 \equiv 2 \bmod{4}\] since our primes are odd. Thus they are all divisible by \(2\) and none by \(4\).

Now assume some odd prime \(p^{\ast}\) divides all of the quantities in question. We'll show no such \(p^{\ast}\) can exist by contradiction.

In much the same way we showed the \(\gcd\) wasn't divisible by \(4\), we seek to find a contradiction in some modulus. But since we are starting with \(p^{2^n} + 1 \equiv 0 \bmod{p^{\ast}}\), if we can find some such \(p\) with \(p \equiv 1 \bmod{p^{\ast}}\), then we'd have our contradiction from
\[0 \equiv p^{2^n} + 1 \equiv 1^{2^n} + 1 \equiv 2 \bmod{p^{\ast}}\] which can't occur since \(p^{\ast}\) is an odd prime.

With this in mind, along with a subsequence of the arithmetic progression \(\left\{5, 13, 21, 29, \ldots\right\}\), it seems that using Dirichlet's theorem on arithmetic progressions may be a good strategy. However, this sequence only tells us about the residue modulo \(8\), but we also want to know about the residue modulo \(p^{\ast}\). Naturally, we look for a subsequence in
\[\mathbb{Z}/\mathbb{8Z} \times \mathbb{Z}/\mathbb{p^{\ast}Z}\] corresponding to the residue pair \((5 \bmod{8}, 1 \bmod{p^{\ast}})\). Due to the Chinese remainder theorem this corresponds to a unique residue modulo \(8p^{\ast}\).

Since this residue \(r\) has \(r \equiv 1 \bmod{p^{\ast}}\), we must have
\[r \in \left\{1, 1 + p^{\ast}, 1 + 2p^{\ast}, \ldots, 1 + 7p^{\ast}\right\} .\] But since \(1 + kp^{\ast} \equiv r \equiv 5 \bmod{8}\), we have \(kp^{\ast} \equiv 4 \bmod{8}\) and \(k \equiv 4\left(p^{\ast}\right)^{-1} \bmod{8}\) since \(p^{\ast}\) is odd and invertible mod \(8\). But this also means its inverse is odd, hence \(k \equiv 4\cdot(2k' + 1) \equiv 4 \bmod{8}\). Thus we have \(1 + 4 p^{\ast} \in \mathbb{Z}/8p^{\ast}\mathbb{Z}\) corresponding to our residue pair. Thus every element in the arithmetic progression \(S = \left\{(1 + 4p^{\ast}) + (8p^{\ast})k \right\}_{k=0}^{\infty}\) is congruent to \(1 + 4 p^{\ast} \bmod{8p^{\ast}}\) and hence \(5 \bmod{8}\) and \(1 \bmod{p^{\ast}}\).

What's more, since \(5 \in \left(\mathbb{Z}/8\mathbb{Z}\right)^{\times}\) and \(1 \in \left(\mathbb{Z}/p^{\ast}\mathbb{Z}\right)^{\times}\), we have \(1 + 4 p^{\ast} \in \left(\mathbb{Z}/8p^{\ast}\mathbb{Z}\right)^{\times}\) (again by the Chinese remainder theorem). Thus the arithmetic progression \(S\) satisfies the hypothesis of Dirichlet's theorem. Hence there must at least one prime \(p\) occurring in the progression (since there are infinitely many). But that also means \(p\) occurs in \(\left\{5, 13, 29, 37, \ldots\right\}\) hence we've reached our desired contradiction. RAA.

Now What?

We still don't know if the strong version \((2)\)
\[\gcd\left(5^{2^n} + 1, 13^{2^n} + 1, \ldots, p^{2^n} + 1, \ldots \right) = 2\] By similar arguments as above, if any odd prime \(p^{\ast}\) divides this \(\gcd\), then we have
\[5^{2^n} \equiv -1 \bmod{p^{\ast}}\] hence there is an element of order \(2^{n + 1}\). This means the order of the multiplicative group \(\varphi\left(p^{\ast}\right) = p^{\ast} - 1\) is divisible by \(2^{n + 1}\). Beyond that, who knows? We're still thinking about it (but only passively, more important things to do).

Sunday, August 18, 2013

Some Fibonacci Fun with Primes

I haven't written in way too long and just wanted to post this fun little proof.

Assertion: Let \(F_n\) be the \(n\)th Fibonacci number defined by \(F_n = F_{n-1} + F_{n-2}\), \(F_0 = 0, F_1 = 1\). Show that for an odd prime \(p\neq 5\), we have \(p\) divides \(F_{p^2−1}\).

Proof: We do this by working inside \(\mathbb{F}_p\) instead of working in \(\mathbb{R}\). The recurrence is given by

\[ \left( \begin{array}{cc}
1 & 1 \\
1 & 0 \end{array} \right)
\left( \begin{array}{c}
F_{n-1} \\
F_{n-2} \end{array} \right)
\left( \begin{array}{c}
F_{n-1} + F_{n-2} \\
F_{n-1} \end{array} \right)
\left( \begin{array}{c}
F_n \\
F_{n-1} \end{array} \right)\] and in general
\[ \left( \begin{array}{cc}
1 & 1 \\
1 & 0 \end{array} \right)^{n}
\left( \begin{array}{c}
1 \\
0 \end{array} \right)
\left( \begin{array}{cc}
1 & 1 \\
1 & 0 \end{array} \right)^{n}
\left( \begin{array}{c}
F_1 \\
F_0 \end{array} \right)
\left( \begin{array}{c}
F_{n + 1} \\
F_n \end{array} \right)\] The matrix \(A = \left(\begin{array}{cc} 1 & 1 \\ 1 & 0 \end{array} \right)\) has characteristic polynomial
\[\chi_A(t) = (1 - t)(0 - t) - (1)(1) = t^2 - t - 1\] If this polynomial has distinct roots, then \(A\) is diagonalizable (this is sufficient, but not necessary). Assuming the converse we have \(\chi_A(t) = (t - \alpha)^2\) for some \(\alpha \in \mathbb{F}_p\); we can assume \(\alpha \in \mathbb{F}_p\) since \(-2\alpha = -1\) is the coefficient of \(t\), which means \(\alpha = 2^{-1} \) (we are fine with this since \(p\) odd means that \(2^{-1}\) exists). In order for this to be a root of \(\chi_A\), we must have
\[0 \equiv 4 \cdot \chi_A\left(2^{-1}\right) \equiv 4 \cdot \left(2^{-2} - 2^{-1} - 1\right) \equiv 1 - 2 - 4 \equiv -5 \bmod{p}.\] Since \(p \neq 5\) is prime, this is not possible, hence we reached a contradiction and \(\chi_A\) does not have a repeated root.

Thus we may write \(\chi_A(t) = (t - \alpha)(t - \beta)\) for \(\alpha, \beta \in \mathbb{F}_{p^2}\) (it's possible that \(\chi_A\) is irreducible over \(\mathbb{F}_p\), but due to degree considerations it must split completely over \(\mathbb{F}_{p^2}\)). Using this, we may write

\[A = P \left(\begin{array}{cc} \alpha & 0 \\ 0 & \beta \end{array} \right) P^{-1}\] for some \(P \in GL_{2} \left(\mathbb{F}_{p^2}\right)\) and so

\[A^{p^2 - 1} = P \left(\begin{array}{cc} \alpha & 0 \\ 0 & \beta \end{array} \right)^{p^2 - 1} P^{-1}
= P \left(\begin{array}{cc} \alpha^{p^2 - 1}  & 0 \\ 0 & \beta^{p^2 - 1}  \end{array} \right)P^{-1}\] Since \(\chi_A(0) = 0 - 0 - 1 \neq 0\) we know \(\alpha\) and \(\beta\) are nonzero, hence \(\alpha^{p^2 - 1} = \beta^{p^2 - 1} = 1 \in \mathbb{F}_{p^2} \). Thus \(A^{p^2 - 1} = P I_2 P^{-1} = I_2\) and so

\[\left( \begin{array}{c}
F_p \\
F_{p^2 - 1} \end{array} \right)
\left( \begin{array}{cc}
1 & 1 \\
1 & 0 \end{array} \right)^{p^2 - 1}
\left( \begin{array}{c}
1 \\
0 \end{array} \right)
I_2 \left( \begin{array}{c}
1 \\
0 \end{array} \right)
\left( \begin{array}{c}
1 \\
0 \end{array} \right)\] so we have \(F_{p^2 - 1} = 0\) in \(\mathbb{F}_p\) as desired.

Monday, December 24, 2012

Bridging OAuth 2.0 objects between GData and Discovery

My colleague +Takashi Matsuo and I recently gave a talk about using OAuth2Decorator (from the google-api-python-client library) with request handlers in Google App Engine. Shortly after, a Stack Overflow question sprung up asking about the right way to use the decorator and, as a follow up,  if the decorator could be used with the Google Apps Provisioning API. As I mentioned in my answer,
The Google Apps Provisioning API is a Google Data API...As a result, you'll need to use the gdata-python-client library to use the Provisioning API. Unfortunately, you'll need to manually convert from a oauth2client.client.OAuth2Credentials object to a gdata.gauth.OAuth2Token object to use the same token for either one.
Instead of making everyone and their brother write their own, I thought I'd take a stab at it and write about it here. The general philosophy I took was that the token subclass should be 100% based on an OAuth2Credentials object:
  • the token constructor simply takes an OAuth2Credentials object
  • the token refresh updates the OAuth2Credentials object set on the token
  • values of the current token can be updated directly from the OAuth2Credentials object set on the token
Starting from the top, we'll use two imports:
import httplib2
from gdata.gauth import OAuth2Token
The first is needed to refresh an OAuth2Credentials object using the mechanics native to google-api-python-client, and the second is needed so we may subclass the gdata-python-client native token class.

As I mentioned, the values should be updated directly from an OAuth2Credentials object, so in our constructor, we first initialize the values to None and then call our update method to actual set the values. This allows us to write less code, because, repeating is bad (I think someone told me that once?).
class OAuth2TokenFromCredentials(OAuth2Token):
  def __init__(self, credentials):
    self.credentials = credentials
    super(OAuth2TokenFromCredentials, self).__init__(None, None, None, None)
We can get away with passing four Nones to the superclass constructor, as it only has four positional arguments: client_idclient_secret, scope,  and user_agent. Three of those have equivalents on the OAuth2Credentials object, but there is no place for scope because that part of the token exchange handled elsewhere (OAuth2WebServerFlow) in the google-api-python-client library.
  def UpdateFromCredentials(self):
    self.client_id = self.credentials.client_id
    self.client_secret = self.credentials.client_secret
    self.user_agent = self.credentials.user_agent
Similarly, the OAuth2Credentials object only implements the refresh part of the OAuth 2.0 flow, so only has the token URI, hence auth_urirevoke_uri and redirect_uri will not be set either. However, the token URI and the token data are the same for both.
    self.token_uri = self.credentials.token_uri
    self.access_token = self.credentials.access_token
    self.refresh_token = self.credentials.refresh_token
Finally, we copy the extra fields which may be set outside of a constructor:
    self.token_expiry = self.credentials.token_expiry
    self._invalid = self.credentials.invalid
Since OAuth2Credentials doesn't deal with all parts of the OAuth 2.0 process, we disable those methods from OAuth2Token that do.
  def generate_authorize_url(self, *args, **kwargs): raise NotImplementedError
  def get_access_token(self, *args, **kwargs): raise NotImplementedError
  def revoke(self, *args, **kwargs): raise NotImplementedError
  def _extract_tokens(self, *args, **kwargs): raise NotImplementedError
Finally, the last method which needs to be implemented is _refresh, which should refresh the OAuth2Credentials object and then update the current GData token after the refresh. Instead of using the passed in request object, we use one from httplib2 as we mentioned in imports.
  def _refresh(self, unused_request):
After refreshing the OAuth2Credentials object, we can update the current token using the same method called in the constructor.

Using this class, we can simultaneously call a discovery-based API and a GData API:
from apiclient.discovery import build
from gdata.contacts.client import ContactsClient

service = build('calendar', 'v3', developerKey='...')

class MainHandler(webapp2.RequestHandler):
  def get(self):
    auth_token = OAuth2TokenFromCredentials(decorator.credentials)
    contacts_client = ContactsClient()
    contacts = contacts_client.get_contacts()
    events ='primary').execute(

Monday, September 10, 2012

Last to Cross the Finish Line: Part Three

Recently, my colleague +Fred Sauer and I gave a tech talk called "Last Across the Finish Line: Asynchronous Tasks with App Engine". This is part three in a three part series where I will share our learnings and give some helpful references to the App Engine documentation.

Check out the previous post if you haven't already. In this section, we'll define the PopulateBatch function and explore the ndb models and Task Queue operations that make it work.


Before defining the models and helper functions in, let's first review the imports:
import json

from google.appengine.api import channel
from google.appengine.ext.deferred import defer
from google.appengine.ext import ndb
Again, we import json and channel for serialization and message passing. We import the defer function from the deferred library to abstract away task creation and take advantage of the ability to "defer" a function call to another thread of execution. Finally, we import ndb as a means for interacting with the App Engine Datastore.

Method Wrapper Built for Tasks

As we saw in the BeginWork handler in part two, units of work are passed to PopulateBatch as 3-tuples containing a method, the positional arguments and the keyword arguments to that method.

In order to keep our task from hanging indefinitely due to unseen errors and to implicitly include the work unit in the batch, we define a wrapper around these method calls:
def AlwaysComplete(task, method, *args, **kwargs):
    method(*args, **kwargs)
  except:  # TODO: Consider failing differently.
As you can see, we catch any and all errors thrown by our method and don't retry the method if it fails. In our example, if the call method(*args, **kwargs) fails, the data won’t be sent through the channel and the given square will not show up in the quilt. However, since we catch these exceptions, the batch will complete and the spinner will disappear with this square still missing.

This part is likely going to be customized to the specific work involved, but for our case, we didn't want individual failures to cause the whole batch to fail. In addition, we implicitly link the work unit with a special type of task object in the datastore.

In the finally section of the error catch, we defer the Complete method on the task corresponding to this work unit. We defer the call to this complete method in order to avoid any errors (possibly from a failed datastore action) that the method may cause. If it were to throw an error, since AlwaysComplete is called in a deferred task, the task would retry and our worker unit would execute (or fail) again, which is bad if our user interface is not idempotent.

Task Model

As we saw above, we need a datastore model to represent tasks within a batch. We start out initially with a model having only one attribute — a boolean representing whether or not the task has completed.
class BatchTask(ndb.Model):
  # Very important that the default value True of `indexed` is used here
  # since we need to query on BatchTask.completed
  completed = ndb.BooleanProperty(default=False)
As we know, we'll need to define a Complete method in order to use the task in AlwaysComplete, but before doing so, we'll define another method which will put the task object in the datastore and pass a unit of work to AlwaysComplete:
  def Populate(self, method, *args, **kwargs):
    kwargs['_transactional'] = True
    defer(AlwaysComplete, self.key, method, *args, **kwargs)
In this Populate method, we first put the object in the datastore transactionally by using the ndb.transactional decorator. By adding the _transactional keyword to the keyword arguments, defer strips away the underscore and creates a transactional task. By doing this
"the task is only enqueued — and guaranteed to be enqueued — if the transaction is committed successfully."
We need this deferred task to be enqueued transactionally for consistency of the completed boolean attribute. The datastore put in Populate uses the default value of False, but after Complete is called we want to set this boolean to True. If this value was not consistent, we may have a race condition that resulted in a completed task in the datastore being marked as incomplete. As we'll see later, we rely on this consistency for a query that will help us determine if our batch is done.

To signal that a unit of work has completed, we define the Complete method on the task object:
  def Complete(self):
    self.completed = True

    batcher_parent = self.key.parent().get()
    defer(batcher_parent.CheckComplete, _transactional=True)
It performs two functions. First, it sets completed to True in a transaction. Second, it retrieves the parent entity of the task object and defers the CheckComplete method on this parent. As we will see in more depth in the PopulateBatch function, we use a special type of batch parent object to create an entity group containing all the worker tasks for the batch. We don't want to check if the batch has completed until the datastore put has succeeded, so we defer the call to call to CheckComplete transactionally, just as we did with AlwaysComplete in the Populate method.

NoteIt may seem that these get calls to retrieve the parent via self.key.parent().get() are using more bandwidth than necessary. However, we are relying here on the power of ndb. Using a combination of instance caching and memcache, most (if not all) of these gets will use the cache and will not incur the cost of a round-trip to the datastore.

Batch Parent Model

Given what we rely on in BatchTask, we need to define a special type of datastore object that will act as the parent entity for a batch. Since we are going to use it to check when a batch is complete, we define the boolean attribute all_tasks_loaded to signal whether or not all worker tasks from the batch have begun. We can use this as a short circuit in our CheckComplete method (or as a guard against premature completion).
class TaskBatcher(ndb.Model):
  all_tasks_loaded = ndb.BooleanProperty(default=False, indexed=False)
To check if a batch is complete, we first determine if all tasks have loaded. If that is the case, we perform an ancestor query that simply attempts to fetch the first worker task in the entity group which has not yet completed. If such a task does not exist, we know the batch has completed, and so start to clean up the task and batch parent objects from the datastore.
  def CheckComplete(self):
    # Does not need to be transactional since it doesn't change data
    session_id =
    if self.all_tasks_loaded:
      incomplete = BatchTask.query(BatchTask.completed == False,
      if len(incomplete) == 0:
        channel.send_message(session_id, json.dumps({'status': 'complete'}))

    channel.send_message(session_id, json.dumps({'status': 'incomplete'}))
We again do the utmost at this step to ensure consistency by using an ancestor query:
"There are scenarios in which any pending modifications are guaranteed to be completely applied...any ancestor queries in the High Replication datastore. In both cases, query results will always be current and consistent."
After checking if a batch is complete, we need to communicate the status back to the client. We'll rely on PopulateBatch to create instances of TaskBatcher with the ID of the session corresponding to the batch as the datastore key. We send a status complete or incomplete message to the client using the session ID for the channel. In order to correctly handle these messages on the client, we'll need to update the onmessage handler (defined in part two) to account for status updates:
socket.onmessage = function(msg) {
  var response = JSON.parse(;
  if (response.status !== undefined) {
  } else {
    var squareIndex = 8*response.row + response.column;
    var squareId = '#square' + squareIndex.toString();
    $(squareId).css('background-color', response.color);
Just as the setStatus method revealed the progress spinner when work began, it will remove the spinner when the status is complete.

We'll next define the CleanUp method that is called when the batch is complete:
  def CleanUp(self):
    children = BatchTask.query(ancestor=self.key).iter(keys_only=True)
This method uses the key from the batch parent to perform another ancestor query and creates an object which can iterate over all the keys of the tasks in the batch. By using the delete_multi function provided by ndb, we are able to delete these in parallel rather than waiting for each to complete. After deleting all the tasks, the batcher deletes itself and clean up is done. Since the TaskBatcher.CheckComplete spawns CleanUp in a deferred task, if the deletes time out, the task will try again until all tasks in the batch are deleted.

As a final method on TaskBatcher, we define something similar to BatchTask.Populate that is triggered after all workers in the batch have been added:
  def Ready(self):
    self.all_tasks_loaded = True
This simply signals that all tasks from the batch have loaded by setting all_tasks_loaded to True and calls CheckComplete in case all the tasks in the batch have already completed. This check is necessary because if all worker tasks complete before all_tasks_loaded is True, then none of the checks initiated by those tasks would signal completion. We use a transaction to avoid a race condition with the initial datastore put — a put which is a signal that all tasks have not loaded.

Populating a Batch

With our two models in hand, we are finally ready to define the PopulateBatch function used (in part two) by the BeginWork handler. We want users of this function to be able to call it directly, but don't want it to block the process they call it in, so we wrap the real function in a function that will simply defer the work:
def PopulateBatch(session_id, work):
  defer(_PopulateBatch, session_id, work)
In the actual function, we first create a TaskBatcher object using the session ID as the key and put it into the datastore using the default value of False for all_tasks_loaded. Since this is a single synchronous put, it blocks the thread of execution and we can be sure our parent is in the datastore before members of the entity group (the task objects) are created.
def _PopulateBatch(session_id, work):
  batcher_key = ndb.Key(TaskBatcher, session_id)
  batcher = TaskBatcher(key=batcher_key)
After doing this, we loop through all the 3-tuples in the passed in batch of work. For each unit of work, we create a task using the batcher as parent and then call the Populate method on the task using the method, positional arguments and keyword arguments provided in the unit of work.
  for method, args, kwargs in work:
    task = BatchTask(parent=batcher_key)
    task.Populate(method, *args, **kwargs)
Finally, to signal that all tasks in the batch have been added, we call the Ready method on the batch parent:
Note: This approach can cause performance issues as the number of tasks grows, since contentious puts within the entity group can cause task completions to stall or retry. I (or my colleagues) will be following up with two posts on the following topics:
  • using task tagging and pull queues to achieve a similar result, but reducing contention
  • exploring ways to extend this model to a hierarchical model where tasks may have subtasks

Wednesday, August 29, 2012

Last to Cross the Finish Line: Part Two

Recently, my colleague +Fred Sauer and I gave a tech talk called "Last Across the Finish Line: Asynchronous Tasks with App Engine". This is part two in a three part series where I will share our learnings and give some helpful references to the App Engine documentation.

Check out the previous post if you haven't already. In this section, we'll cover the two WSGI handlers in serving requests for our application and the client side code that communicates with our application.


Before defining the handlers, let's first review the imports:
import json

from google.appengine.api import channel
from google.appengine.api import users
from google.appengine.ext.webapp.util import login_required
import webapp2
from webapp2_extras import jinja2

from display import RandomRowColumnOrdering
from display import SendColor
from models import PopulateBatch
We import json for serialization of messages. Specific to App Engine, we import channel to use the Channel API, users and login_required for authenticating users within a request, webapp2 for creating WSGI Handlers and jinja2 for templating.

Finally, we import four functions from the two other modules defined within our project. From the display module, we import the SendColor function that we explored in part one and the RandomRowColumnOrdering function, which generates all possible row, column pairs in a random order. From the as of yet undiscussed models module we import the PopulateBatch function, which takes a session ID and a batch of work to be done and spawns workers to carry out the batch of work.


This module defines two handlers: the main page for the user interface and an AJAX handler which will begin spawning the workers.

For the main page we use jinja2 templates to render from the template main.html in the templates folder:
class MainPage(webapp2.RequestHandler):
  def RenderResponse(self, template, **context):
    jinja2_renderer = jinja2.get_jinja2(
    rendered_value = jinja2_renderer.render_template(template, **context)
  def get(self):
    user_id = users.get_current_user().user_id()
    token = channel.create_channel(user_id)
    self.RenderResponse('main.html', token=token, table_id='pixels',
                        rows=8, columns=8)
In get — the actual handler serving the GET request from the browser — we use the login_required decorator to make sure the user is signed in, and then create a channel for message passing using the ID of the signed in user. The template takes an HTML ID, rows and columns to create an HTML table as the "quilt" that the user will see. We pass the created token for the channel, an HTML ID for the table and the rows and columns to the template by simply specifying them as keyword arguments.

For the handler which will spawn the workers, we use RandomRowColumnOrdering to generate row, column pairs. Using each pair along with the SendColor function and the user ID (as a proxy for session ID) for message passing, we add a unit of work to the batch
class BeginWork(webapp2.RequestHandler):
  # Can't use login_required decorator here because it is not
  # supported for POST requests
  def post(self):
    response = {'batch_populated': False}
      # Will raise an AttributeError if no current user
      user_id = users.get_current_user().user_id()
      # TODO: return 400 if not logged in 
      work = []
      for row, column in RandomRowColumnOrdering(8, 8):
        args = (row, column, user_id)
        work.append((SendColor, args, {}))  # No keyword args
      PopulateBatch(user_id, work)
      response['batch_populated'] = True
      # TODO: Consider logging traceback.format_exception(*sys.exc_info()) here
Finally, for routing applications within our app, we define:
app = webapp2.WSGIApplication([('/begin-work', BeginWork),
                               ('/', MainPage)],
and specify
- url: /.*
in app.yaml; to use WSGI apps, the App Engine runtime must be python27.

Client Side Javascript and jQuery

In the template main.html we use jQuery to make AJAX requests and manage the CSS for each square in our "quilt". We also define some other Javascript functions for interacting with the App Engine Channel API. In the HTML <head> element we load the Channel Javascript API, and in the <body> element we open a channel using the {{ token }} passed in to the template:
  <script src="/_ah/channel/jsapi"></script>
  <script type="text/javascript">
    channel = new goog.appengine.Channel('{{ token }}');
    socket =;
    socket.onerror = function() { console.log('Socket error'); };
    socket.onclose = function() { console.log('Socket closed'); };
In addition to onerror and onclose, we define more complex functions for the onopen and onmessage callbacks.

First, when the socket has been opened, we send a POST request to /begin-work to signal that the channel is ready for communication. If the response indicates that the batch of workers has been initialized successfully, we call a method setStatus which will reveal the progress spinner:
socket.onopen = function() {
  $.post('/begin-work', function(data) {
    var response = JSON.parse(data);
    if (response.batch_populated) {
      setStatus('Loading began');
As we defined in part one, each SendColor worker sends back a message along the channel representing a row, column pair and a color. On message receipt, we use these messages to set the background color of the corresponding square to the color provided:
socket.onmessage = function(msg) {
  var response = JSON.parse(;
  var squareIndex = 8*response.row + response.column;
  var squareId = '#square' + squareIndex.toString();
  $(squareId).css('background-color', response.color);
As you can see from squareId, each square in the table generated by the template has an HMTL ID so we can specifically target it.


In the final post, we'll define the PopulateBatch function and explore the ndb models and Task Queue operations that make it work.

Monday, August 27, 2012

Last to Cross the Finish Line: Part One

Recently, my colleague +Fred Sauer and I gave a tech talk called "Last Across the Finish Line: Asynchronous Tasks with App Engine". This is part one in a three part series where I will share our learnings and give some helpful references to the App Engine documentation.


Before I dive in, a quick overview of our approach:
  • "Fan out; Fan in" First spread tasks over independent workers; then gather the results back together
  • Use task queues to perform background work in parallel
    • Tasks have built-in retries
    • Can respond quickly to the client, making UI more responsive
  • Operate asynchronously when individual tasks can be executed independently, hence can be run concurrently
    • If tasks are too work intensive to run synchronously, (attempt to) break work into small independent pieces
  • Break work into smaller tasks, for example:
    • rendering media (sounds, images, video)
    • retrieving and parsing data from an external service (Google Drive, Cloud Storage, GitHub, ...)
  • Keep track of all workers; notify client when work is complete
Before talking about the sample, let's check it out in action:
We are randomly generating a color in a worker and sending it back to the client to fill in a square in the "quilt". (Thanks to +Iein Valdez for this term.) In this example, think of each square as a (most likely more complex) compute task.

Application Overview

The application has a simple structure:
|-- app.yaml
+-- templates/
       +-- main.html
We'll inspect each of the Python modules, and individually and explore how they interact with one another. In addition to this, we'll briefly inspect the HTML and Javascript contained in the template main.html, to understand how the workers pass messages back to the client.

In this post, I will explain the actual background work we did and briefly touch on the methods for communicating with the client, but won't get into client side code or the generic code for running the workers and watching them all as they cross the finish line. In the second post, we’ll examine the client side code and in the third, we’ll discuss the models that orchestrate the work.


These worker methods are defined in To generate the random colors, we simply choose a hexadecimal digit six different times and throw a # on at the beginning:
import random

HEX_DIGITS = '0123456789ABCDEF'

def RandHexColor(length=6):
  result = [random.choice(HEX_DIGITS) for _ in range(length)]
  return '#' + ''.join(result)
With RandHexColor in hand, we define a worker that will take a row and column to be colored and a session ID that will identify the client requesting the work. This worker will generate a random color and then send it to the specified client along with the row and column. To pass messages to the client, we use the Channel API and serialize our messages using the json library in Python.
import json
from google.appengine.api import channel

def SendColor(row, column, session_id):
  color = RandHexColor(length=6)
  color_dict = {'row': row, 'column': column, 'color': color}
  channel.send_message(session_id, json.dumps(color_dict))


In the next post, we'll explore the WSGI handlers that run the application and the client side code that handles the messages from the workers.