Showing posts with label theory. Show all posts
Showing posts with label theory. Show all posts

Obama on sorting 1M integers: Bubble sort the wrong way to go

Recently StackOverflow and Hacker News discussed the question of how to sort 1 million 8-digit numbers in 1 megabyte of RAM. This reminded me of when I saw Obama in 2007 at Google - Eric Schmidt asked Obama how to sort 1 million integers as a laugh line, but Obama shocked him by answering "I think the bubble sort would be the wrong way to go." The video is pretty entertaining:

Here's the transcript:

SCHMIDT: Now, Senator, you're here at Google and I like to think of the presidency as a job interview. Now, it's hard to get a job as president. And--I mean, you're going to do a great job. It's also hard to get a job at Google. We have questions and we ask our candidates questions. And this one is from Larry Schwimmer. What--you guys think I'm kidding, it's right here.[1] What is the most efficient way to sort a million 32-bit integers?

OBAMA: Well...

SCHMIDT: Maybe--I'm sorry...

OBAMA: No, no, no, no. I think--I think the bubble sort would be the wrong way to go.

SCHMIDT: [facepalm] Come on. Who told him this? Okay. I didn't see computer science in your background.

OBAMA: We've got our spies in there.[2]

If you're wondering about the complete answer to the sorting question, see a detailed explanation.[3]

All in all, it was a very unexpected answer from Obama with perfect delivery.

Notes

[1] Nervous laughter greeted the mention of a Larry Schwimmer question, because he once asked Jimmy Carter about his UFO encounter.

[2] Eric Schmidt had asked the sorting question to John McCain on a different visit, with the expected result. See the YouTube clip of McCain's visit.

[3] The pedantic may note that Obama's question is slightly different from the Stack Overflow question since it involves 32-bit integers vs. 8 digit integers, and the memory constraint was omitted.

My Knuth reward check

I attended a very interesting talk "All Questions Answered" by the famous computer science professor Don Knuth in March, where he talked about many things, including his new book.

The talk inspired me to read The Art of Computer Programming, Volume 4A: Combinatorial Algorithms. As he described in the talk, he gives a reward check (for 1 hexadecimal dollar) to anyone who finds an error in one of his books, so I set myself the goal of finding an error in the book while on vacation.

After a lot of study, I thought I'd found an error in a diagram, but then I realized I was confused. Next, I found a blatant error in an appendix, but that error had already been discovered and was listed in the errata. Finally I found an error on page 574 that I hoped hadn't been found yet and sent it off to Professor Knuth.

I was delighted to receive my reward check today, which Wikipedia calls "among computerdom's most prized trophies". That's a big exaggeration but nonetheless, I'm happy to get one. Note that the check is from the fictional Bank of San Seriffe to avoid check fraud problems from all the images of his checks on the web.

My Knuth TAOCP reward check

As for the book itself, it's interesting even if you're not after a reward, although very challenging to read. Volume 4a describes combinatorial algorithms, including more ways of computing permutations and combinations than you can shake a stick at. It also has an extensive discussion of BDDs and ZDDs, which are newish data structures for representing sets. The section on bitwise tricks and techniques is interesting if you like HAKMEM-style tricks such as reversing the bits in an integer through a short sequence of incomprehensible operations.

I have to admit that trying to find an error in a book is a strange vacation goal, but I'm glad I succeeded and Knuth's book taught me some interesting algorithms in the process.

P.S. I was very surprised to see this article on the front page of Hacker News. To answer the questions there and below, the error I found was in volume 4a page 574 (as the memo line on the check shows). The solution to exercise 67 on that page says a particular circuit uses 6 ANDN gates, which I thought should be NAND gates. It gets a bit more complicated because Knuth was referring to the ANDN op code for MMIX, but there was still a mistake with and-not versus not-and. (The other error I noticed was n choose k on page 824, but I checked the errata and it had already been found.)

The Y Combinator in Arc and Java

I was recently reading The Little Schemer, a very intersting book about how to think in Scheme. The book uses a unique question-response style to present simple concepts such as cons and recursion, as well as complex concepts such as using lambda functions to build arithmetic out of fundamental axioms, and deriving the Y Combinator.

As you're probably aware, the Y Combinator is a fixed-point combinator, which lets you build recursion out of an anonymous, non-recursive function. The tricky part of building recursion without naming is that if you can't name a function, how can the function call itself? The Y Combinator is a way to do this.

In Arc, it's easy to write a recursive factorial function:

(def fact (n)
  (if (is n 0) 1
      (* n (fact (- n 1)))))
Note that def assigns the name fact to this function, allowing it to recursively call itself. But without def, how can the function call itself?

Fixed point as an infinite stack of function calls

Let's make a function fact-gen that if passed the factorial function as input, returns the factorial function. (Note that fn is Arc's equivalent of lambda.)
(def fact-gen (fact-in)
 (fn (n)
  (if (is n 0) 1
      (* n (fact-in (- n 1))))))
This may seem rather useless, returning the factorial function only if you already have it. Since we don't have a factorial function to pass in, we'll pass in nil (technically bottom). This at least gets us started:
arc> ((fact-gen nil) 0)
1
arc> ((fact-gen nil) 1)
Error: "Function call on inappropriate object nil (0)"
We can compute factorial of 0, but factorial of 1 hits nil and dies. But we can take our lame factorial function, pass it in to fact-gen and get a slightly better factorial function that can compute factorials up to 1:
arc> ((fact-gen (fact-gen nil)) 1)
1
We can repeat this to get an even more useful factorial function:
arc> ((fact-gen (fact-gen (fact-gen (fact-gen (fact-gen nil))))) 4)
24
If we could have an infinite stack of fact-gen calls, then we would actually have the factorial function. But we can't do that. Or can we?

The fixed point of a function f is a value x for which f(x) = x. We can apply the same idea to our infinite stack of fact-gen. Since applying fact-gen to the infinite stack one more time makes no difference, (fact-gen infinite-stack) = infinite-stack, so the infinite stack is a fixed point of fact-gen. Thus, the fixed-point of fact-gen is the factorial function.

If taking the fixed point of an algorithmic function seems dubious to you, a full explanation is available in Chapter 5 of the 1300 page tome Design Concepts in Programming Languages; trust me, it's all rigorously defined.

The fixed-point combinator

So how do you find the fixed point without an infinite stack of functions? That's where the fixed-point combinator (also known as the Y combinator) comes in. The fixed-point combinator Y takes a function and returns the fixed point of the function. That is, applying the function once more makes no difference:
y f = f (y f)
You may wonder how the Y combinator computes an infinite stack of functions. The intution is it computes a finite stack that is just big enough for the argument.

The fixed-point combinator in Haskell

In Haskell, you can use the above definition of the Y combinator directly:
y(f) = f (y f)
fact f n = if (n == 0) then 1 else n * f (n-1)
y(fact) 10
This is a bit of a "cheat", since the definition of the y combinator takes advantage of Haskell's pre-existing recursion, rather than providing recursion from scratch. Note that this only works because of lazy evalation; otherwise the definition of y is an infinite loop. (Haskell includes the Y combinator under the name fix.)

The Y combinator in Arc

The Little Schemer derives the Y combinator in Scheme. The Arc version is very similar:
(def Y (r)
  ((fn (f) (f f))
   (fn (f)
     (r (fn (x) ((f f) x))))))
If the Y combinator is applied to the earlier fact-gen, it yields a recursive factorial function. Like magic:
arc>((Y fact-gen) 10)
3628800

You may protest that this doesn't really implement anonymous recursion since both Y and fact-gen are explicitly named with def, so you could really just call fact-gen directly. That naming is just for clarity; the whole thing can be done as one big anonymous function application:

arc> (((fn (r)
  ((fn (f) (f f))
   (fn (f)
     (r (fn (x) ((f f) x))))))

(fn (fact)
 (fn (n)
  (if (is n 0) 1
      (* n (fact (- n 1)))))))

10)
3628800
Now you can see the recursive factorial can be computed entirely with anonymous functions, not a def in sight. The first blob is the Y combinator; it is applied to the second blob, the factorial generator, and the resulting function (factorial) is applied to 10, yielding the answer.

Y Combinator in Java

The Y combinator in a Lisp-like language is not too tricky. But I got to wondering if it would be possible to implement it in Java. I'd done crazy continuation stuff in Java, so why not the Y combinator?

Several objections come to mind. Java doesn't have first-class functions. Java doesn't have closures. Everything in Java is an object. Java is statically typed. Is the idea of a Y combinator in Java crazy? Would it require total Greenspunning?

To implement the Y combinator in Java, I did several things. Since Java doesn't have first-class functions, I wrapped each function in an anonymous inner class with a single method apply(), which executes the function. That is, I used a function object or functor. Since "objects are a poor man's closures" (Norman Adams), I used this object creation in place of each closure. In order to define types, I restricted my Java Y combinator to integer functions on integers. Each type defines an interface, and each object implements the appropriate interface.

Using these techniques, I was able to fairly directly implement the Y combinator in Java. The first part defines a bunch of types: IntFunc is a simple function from integers to integers. IntFuncToIntFunc is the type of the factorial generator, taking an integer function and returning another integer function. FuncToIntFunc is the somewhat incomprehensible type of the Y combinator subexpressions that apply f to f yielding an integer function. Finally, the Y combinator itself is an IntFuncToIntFuncToIntFunc, taking an IntFuncToIntFunc (fact-gen) as argument and returning an IntFunc (the factorial function itself).

class YFact {
  // Integer function returning an integer
  // int -> int
  interface IntFunc { int apply(int n); }

  // Function on int function returning an int function
  // (int -> int) -> (int -> int)
  interface IntFuncToIntFunc { IntFunc apply(IntFunc f); };

  // Higher-order function returning an int function
  // F: F -> (int -> int)
  interface FuncToIntFunc { IntFunc apply(FuncToIntFunc x); }

  // Function from IntFuntToIntFunc to IntFunc
  // ((int -> int) -> (int -> int)) -> (int -> int)
  interface IntFuncToIntFuncToIntFunc { IntFunc apply(IntFuncToIntFunc r);};
Next comes the meat. We define the Y combinator, apply it to the factorial input function, and apply the result to the input argument. The result is the factorial.
  public static void main(String args[]) {
    System.out.println(
      // Y combinator
      (new IntFuncToIntFuncToIntFunc() { public IntFunc apply(final IntFuncToIntFunc r) {
      return (new FuncToIntFunc() {public IntFunc apply(final FuncToIntFunc f) {
          return f.apply(f); }})
 .apply(
          new FuncToIntFunc() { public IntFunc apply(final FuncToIntFunc f) {
         return r.apply(
                new IntFunc() { public int apply(int x) {
    return f.apply(f).apply(x); }});}});}}

    ).apply(
        // Recursive function generator
        new IntFuncToIntFunc() { public IntFunc apply(final IntFunc f) {
          return new IntFunc() { public int apply(int n) {
     if (n == 0) return 1; else return n * f.apply(n-1); }};}} 

    ).apply(
      // Argument
      Integer.parseInt(args[0])));
  }
}
The result is the factorial of the input argument: (source code)
$ javac YFact.java
$ java YFact 10
3628800
Surprisingly, this code really works, implementing the Y combinator. Note that there are no variables (apart from arguments), and no names are assigned to any of the anonymous functions. Yet, we have recursion.

The Java version is considerably more verbose than the Arc version, since each function becomes an object creation wrapping an anonymous function declaration, with a liberal sprinkling of type declarations, public and final. Even so, there is a direct mapping between the Arc code and the Java code. There's no Greenspunning in there, no Lisp simulation layer. Ironically, the Java code starts to look like Lisp code, except with a bunch of }}} instead of ))).

To convince you that the Java recursion works even in a more complex case, we can implement Fibonacci numbers by simply replacing the input function: (source code)

...
        // Recursive Fibonacci input function
        new IntFuncToIntFunc() { public IntFunc apply(final IntFunc f) {
          return new IntFunc() { public int apply(int n) {
            if (n == 0) return 0;
            else if (n == 1) return 1;
            else return f.apply(n-1) + f.apply(n-2); }};}}
...
The code recursively generates Fibonacci numbers:
$ java YFib 30
832040

Is this the "real" Y Combinator?

The typical form of the Y combinator is:
λf.(λx.f (x x)) (λx.f (x x))
and you may wonder why the Y combinator in Arc and Java is slightly different. Because Java (and Scheme, Arc, etc.) are call-by-value languages and not call-by-name languages, they require the applicative-order Y combinator. This combinator has the form:
λr.(λf.(f f)) λf.(r λx.((f f) x))
The call-by-name Y combinator will go into an infinite loop in a call-by-value language. I found this out the hard way, when I implemented the "wrong" Y combinator in Java and quickly got a stack overflow.

For details on applicative-order, eta-reduction, why different combinators are required, and a derivation of the Y combinator, see Sketchy Lisp.

Java vs. Lambda Calculus

In the Java code, new takes the place of λ and apply explicitly shows application, which is implicit in lambda calculus. To make the connection between the Java code and the lambda expression clearer, I have highlighted the key parts of the Java Y combinator:
      // Y combinator
      (new IntFuncToIntFuncToIntFunc() { public IntFunc apply(final IntFuncToIntFunc r) {
      return (new FuncToIntFunc() {public IntFunc apply(final FuncToIntFunc f) {
          return f.apply(f); }})
 .apply(
          new FuncToIntFunc() { public IntFunc apply(final FuncToIntFunc f) {
         return r.apply(
                new IntFunc() { public int apply(int x) {
    return f.apply(f).apply(x); }});}});}}
Note the exact correspondence of the highlighted parts with the lambda calculus expression:
λr.(λf.(f f)) λf.(r λx.((f f) x))

Conclusion

It is possible to implement the Y combinator in Java, showing that Java has more power than many people realize. On the other hand, the Java code is ugly and bulky; a Lisp-like language is a much more natural fit. For more fun, try going through SICP in Java.

Postscript

I received some great feedback with interesting links. Apparently a number of people enjoy implementing the Y combinator in a variety of languages:

Continuations made difficult

For a challenge, I translated the "mondo-bizarro" Arc Continuation Puzzle into Java.

Why would I do that? Because I can :-) But seriously, using continuations in a language entirely unsuited for it is a good way to experience the tradeoffs of different styles of languages, as well as a way to learn more about how continuations work.

This code is entirely different from my Arc version, mainly because in the Arc version I decided to see if the throw/catch idiom could replace call/cc; the Java code is much closer to the original Scheme. Because Java doesn't have first-class continuations, I used the Continuation-passing style of explicitly passing continuations around. Then call/cc is simply replaced with use of the current continuation.

Because Java doesn't support first-class functions, every continuation function needs to be wrapped in a class that implements an interface, which I unimaginatively call Continuation. The "let" also turns into an object creation, resulting in another class. This results in a fair bit of boilerplate to handle all these classes compared to the Scheme code, but the Java code maps recognizably onto the Scheme code.

On the whole, mondo-bizarro worked better in Java than I expected; no Greenspunning required. It produces the expected 11213 output, proving it works. I think the Java code is actually easier to understand, since everything is explicit.

I have also found it entertaining to implement some of the complex SICP exercises in Java; maybe I'll post details later.

(The title of this article is, of course, a homage to Mathematics Made Difficult.)

Here's the code.

/**
 * Mondo-Bizarro ported to Java.
 * Based on mondo-bizarro by Eugene Kohlbecker
 * ACM SIGPLAN Lisp Pointers, Volume 1, Issue 2 (June-July 1987) pp 22-28
 */

/* Original Scheme code:
(define mondo-bizarro
  (let (
        (k (call/cc (lambda (c) c)))
        )
    (write 1)
    (call/cc (lambda (c) (k c)))
    (write 2)
    (call/cc (lambda (c) (k c)))
    (write 3)))
*/


interface Continuation {
  public void run(Continuation c);
}

public class Mondo implements Continuation {

  public static void main(String argv[]) {
    Continuation c = new Mondo();
    c.run(c);
  }

  public void run(Continuation c) {
    Continuation let = new Let();
    let.run(c);
  }
  
  class Let implements Continuation {
    Continuation k;

    public void run(Continuation c) {
      k = c;
      System.out.println("1");
      k.run(new C2());
    }

    class C2 implements Continuation {
      public void run(Continuation c) {
        System.out.println("2");
        k.run(new C3());
      }
    }

    class C3 implements Continuation {
      public void run(Continuation c) {
        System.out.println("3");
      }
    }
  }
}

Continue reading...