Counting Words VI: Clojure

We could do this:

(->> m
     (mapv #(s/join " " %))
     (s/join "\n")
     (#(str % "\n")))

or this:

(-> (mapv #(s/join " " %) m)
    (#(s/join "\n" %))
    (str "\n"))

but they look a bit confusing.

We could do this:

(->> m
     (mapv (comp #(str % "\n")
                 #(s/join " " %)))
     (s/join))

which maps it with a composed lambda.

It means: join with a space every item in this vector of vectors, and add a trailing newline to the result. Then join these strings. Example:

(->> [["a" "b" "c"]
      ["d" "e" "f"]]
     (mapv (comp #(str % "\n")
                 #(s/join " " %)))
     (s/join))
=> "a b c\nd e f\n"

(->> '(("a" "b" "c")
       ("d" "e" "f"))
     (-map (-compose (-cut concat <> "\n")
                     (-cut s-join " " <>)))
     (s-join ""))
=> "a b c\nd e f\n"

(->> '(("a" "b" "c")
       ("d" "e" "f"))
     (--map (concat (s-join " " it) "\n"))
     (s-join ""))
=> "a b c\nd e f\n"

(->> '(("a" "b" "c")
       ("d" "e" "f"))
     (--map (-> (s-join " " it)
                (concat "\n")))
     (s-join ""))
=> "a b c\nd e f\n"

Yet I feel that this is simpler and more readable:

(->> m
     (mapv (fn [[k v]] (format "%s %s\n" k v)))
     s/join)

It means: destructure each pair as key and value, then format it as string using a space and trailing newline, then join all pairs.

So let’s go with that one. Or rather, let’s use a small variation — using str instead of format:

(str k " " v "\n")

So here’s a first solution. It takes a string as its argument, so for now we’re reading it all into memory.

(defn cw-count-it-S1 [s]
  (->> (s/split s #"\s+")
       (mapv s/lower-case)
       (frequencies)
       (cw-make-string)))

That’s all.

1. First, we split the string at whitespace. We now have a vector of individual words. They possibly have punctuation, and removing them would be simple, but we’re just following the exercise’s tokenization rules.
   
2. Then we lowercase everything.
   
3. Then we run frequencies on it — and that function does all the heavy-lifting of counting the words and putting them into a map. It’s part of the standard library.
   
4. Then we run cw-make-string, which removes the "" key, sorts it using our comparator, and converts the map to a final string.
   

A few notes:

• We could have lowercased before splitting rather than after:
  

  (->> s s/lower-case
       (#(s/split % #"\s+"))
       …)
  

• If you, Emacs Lisper, ever wondered what inspired Dash’s threading macros, that’s what. They’re idiomatic and widely used in Clojure, and as much of a joy to code with as they are in dash.el.
  
• Newer versions of dash.el have -frequencies, in case you need this in Elisp. But unlike Clojure, this function isn’t part of Elisp’s core libraries, because dash isn’t.
  

• The #(...%...) thing is a short and sweet way to express lambdas (which Clojure calls fns). In Clojure, these are native syntax. In Elisp, you can get something closely resembling it — but only by using a macro from a third-party library, because Emacs core didn’t accept #(...%...) as actual syntax.
  

  But even if you use it, you wouldn't be able to thread it directly in Elisp as you can in Clojure.

  (-> "Foo bar QUUX" downcase
      ((##s-split "[ \n\r\t]+" %)))
  !!> invalid-function
  

  You’d need to use the whole explicit lambda instead:
  

  (-> "Foo bar QUUX" downcase
      ((lambda (s) (s-split "[ \n\r\t]+" s))))
  => '("foo" "bar" "quux")
  

  But this syntax is discouraged, so if you try to compile it you’ll get a warning:
  

  Warning: Use of deprecated ((lambda (x) ...) ...) form
  

  “Ah!”, you think, “I’ll just use the newer lambda literal that came with Emacs 30” — so you try this and it works!
  

  (-> "Foo bar QUUX" downcase
      (#[(s) ((s-split "[ \n\r\t]+" s)) (t)]))
  => '("foo" "bar" "quux")
  

  But apart from the fact that the new literal is just as long as the old lambda was, and noisy, you’ll still get a compiler warning — a different one:
  

  Warning: Use ‘funcall’ instead of ‘#[(s) ((s-split "[ \n\r\t]+" s)) (t)]’ in the function position
  

  by which they mean that although this sort of thing works:
  

  ((lambda (x) (+ x 7)) 35) => 42
  

  it’s been deprecated (I don’t know when or why), so you have to add a funcall:
  

  (funcall (lambda (x) (+ x 7)) 35) => 42
  

  which, besides being long, wouldn’t work for our case because we’re first-threading.
  

  You could last-thread it to get rid of the lambda:
  

  (->> "Foo bar QUUX" downcase
       (s-split "[ \n\r\t]+"))
  => '("foo" "bar" "quux")
  

  but mixed threading positions is the reason why we’re using a lambda here in the first place: it’d fix this one but break others.
  

  So how do you mix-thread in Elisp? You could either thread anaphorically, with -->:
  

  (--> "Foo bar QUUX" downcase
       (s-split "[ \n\r\t]+" it))
  => '("foo" "bar" "quux")
  

  or use -as->, which is the same as Clojure’s as->:
  

  (-as-> "Foo bar QUUX" myvar
         (downcase myvar)
         (s-split "[ \n\r\t]+" myvar))
  => '("foo" "bar" "quux")
  

  whereas in Clojure you can either use as->:
  

  (as-> "Foo bar QUUX" myvar
    (s/lower-case myvar)
    (s/split myvar #"[ \n\r\t]+"))
  => '["foo" "bar" "quux"]
  

  (which has the downside of having to repeat the variable name at every step), or use “fn-sexp-as-first” (“lambda-sexp-as-car”) for the exceptions:
  

  (->> "Foo bar QUUX"
       (s/lower-case)
       (#(s/split % #"[ \n\r\t]+")))
  

  which is doing this:
  

  (#(s/split % #"[ \n\r\t]+") "foo bar quux") => '["foo" "bar" "quux"]
  

  which, were we not last-threading, would be more simply written as this:
  

  (s/split "foo bar quux" #"[ \n\r\t]+") => '["foo" "bar" "quux"]
  

  but we are last-threading, and that puts the last result in the first position.
  

  Using “fn-sexp-as-first” gives you flexibility to put the variable anywhere inside the fn: since the whole thing receives only one arg, whatever comes can, flexibly, be either first or last.
  

  Ok, enough threading-related digressions.
  

Time to put back one of the constraints: we’ll now read the string from a file — but still read it all into memory.

We’ll also write the result to a file.

(defn cw-count-it-F1 [file-in file-out]
  (->> (slurp file-in)
       (#(s/split % #"\s+"))
       (mapv s/lower-case)
       (frequencies)
       (cw-make-string)
       (spit file-out)))

And then:

(def fooin "/tmp/count/foo.txt")
(def fooF1 "/tmp/count/fooF1.txt")
(cw-count-it-F1 fooin fooF1)

It works:

cat /tmp/count/fooF1.txt

bar, 3
the 2
12 1
bar: 1
bars' 1
foo's 1
foo. 1
foozle 1

The only changes were:

• Instead of a string, the function now takes two args: filename of input, filename of output.
  
• s changed to (slurp file-in), and we added a (spit file-out) at the end.
  

Just slurp and spit.

(-> (f-read file-in)
    do-stuff-with-string
    (f-write 'utf-8 file-out))

(let ((s (with-temp-buffer
           (insert-file-contents file-in)
           (buffer-string))))
  (setq s (do-stuff-with-string s))
  (write-region s nil file-out))

(with-temp-buffer
  (insert-file-contents file-in)
  (do-stuff-in-buffer)
  (write-region (point-min) (point-max) file-out))

(while (re-search-forward "some-REGEXP" nil t)
  (push (match-string) ...)

So now let’s re-introduce the last remaining constraint: read the file line by line instead of slurping it all into memory.

(defn cw-count-it-L1 [file-in file-out]
  (with-open [rdr (jio/reader file-in)]
    (binding [*in* rdr]
      (loop [ln (read-line), m {}]
        (if ln
          (recur (read-line) (->> (s/split (s/lower-case ln) #"\s+")
                                  (frequencies)  (merge-with + m)))
          (->> m (cw-make-string) (spit file-out)))))))

I read this as follows:

Bind standard input to a reader of the input file. Create an empty map, and while there are lines to read, read one of them, lowercase it, and split it at whitespace. We’ll have words (with possible punctuation). Get their frequencies into a map. Merge this map with our accumulating map (which started empty). When the merge involve keys that show up in both maps, add their values.

When we run out of lines to read, pick the accumulated map and run cw-make-string — which removes the "" key, sorts by count descending and alphabetically when there’s a tie, then replaces every pair with a string: key space value newline, and joins these strings. Write the result to file-out.

If we test it:

(def fooin "/tmp/count/foo.txt")
(def fooL1 "/tmp/count/fooL1.txt")
(cw-count-it-L1 fooin fooL1)

we get the same result as the previous F1:

cat /tmp/count/fooL1.txt

bar, 3
the 2
12 1
bar: 1
bars' 1
foo's 1
foo. 1
foozle 1

only that now there’s a trailing newline and we’re reading the file line by line.

Running this over the actual large input data file produces the expected result: the hashes match.

Now, I think the L1 above is absolutely fine, and simple, and good enough for the problem at hand.

Yet you could say “But flandrew: isn’t this one of those pleasingly parallel problems? Couldn’t we, you know, do all that counting in parallel?”.

And you’d have a point. For example, we could do something like this:

(defn cw-count-it-P0 [file-in file-out]
  (with-open [rdr (jio/reader file-in)]
    (->> rdr  line-seq  (partition-all 10000)
         (pmap (fn [part]
                 (reduce #(merge-with
                           + %1 (frequencies
                                 (s/split (s/lower-case %2) #"\s+")))
                         {} part)))
         (apply merge-with +)
         (cw-make-string)
         (spit file-out))))

or this:

(defn cw-count-it-P1 [file-in file-out]
  (with-open [rdr (jio/reader file-in)]
    (->> rdr  line-seq  (partition-all 10000)
         (pmap #(loop [lns % m {}]
                  (if-let [ln (first lns)]
                    (recur (rest lns)
                           (->> (s/split (s/lower-case ln) #"\s+")
                                (frequencies)  (merge-with + m)))
                    m)))
         (apply merge-with +)
         (cw-make-string)
         (spit file-out))))

which I read as follows:

Bind standard input to a reader of the input file. Create a lazy sequence of all the lines of the file. Partition this sequence in chunks of 10000 lines.

Now process these lazy 10000-line chunks in parallel. For each of them, loop over its lines sequentially, and keep accumulating the word counts in a map.

After all chunks have been processed, return their maps, merge them, run cw-make-string — which removes the "" key, sorts by count descending and alphabetically when there’s a tie, then replaces every pair with a string: key space value newline, and joins these strings.

And if we test it:

(def fooin "/tmp/count/foo.txt")
(def fooP1 "/tmp/count/fooP1.txt")
(cw-count-it-P1 fooin fooP1)

we get the same results.

Likewise with the actual huge input file provided by the exercise.

Whether this will be faster or not, and by how much, depends on your machine — but also on the chunking size, because parallelizing itself adds an overhead. Frankly, for this particular exercise and its input size (not that huge), I’d stick with the simpler L1.