moved!

moved to wordpress.com

http://ujihisa.wordpress.com/
see you there!

if_lua accelerates Vim scripts

Vim has some foreign language interfaces available for Vim script, to allow you to make a Vim plugin with using other programming languages besides Vim script, such as Python, Scheme, or Ruby. This is not only handy that you can attempt to make your code more readable with using ways of writing code that Vim script can't or using libraries exclusive for the language, but also provides faster computation. Vim script runs slower than many of you expect.

Lua is a general purpose programming language which works particularly well about embedding to an existing system written in C. It is very natural that Vim has lua interface as if_lua.

The significant part of if_lua is that it works really fast.

Comparison between Pure Vim script, if_lua, and if_python: https://github.com/vim-jp/issues/issues/48#issuecomment-15355516

• Vim script: 1.4985sec
• Luajit: 0.0013sec (1152 times faster than Vim script)
• Python: 0.0272sec (20 times faster than Vim script)

Time to start

• if_lua = 0.000297sec
• if_python = 0.016656sec
• if_ruby = 0.039039sec

Now it's default!

gentoo

I submitted a patch to Gentoo Linux 10 month ago, and it was accepted finally in a couple days ago!

• https://bugs.gentoo.org/show_bug.cgi?id=436744 Gentoo's Bugzilla – Bug 436744 vim.eclass - add lua support

So if you simply emerge vim or gvim with adding lua and luajit USE flags, the vim will have lua integration!

FAQ

• Does speed matter? My Vim already works fast enough.
  • Speed matters. Some auto-completion plugins gave up providing powerful functionality since it took too much time. If it's fast enough that users can't recognize, the authors can implement more functionalities.
• Python is slower than Luajit and Lua, but it's more popular among Vim plugin authors. We should keep using Python.
  • Popularity is just a chicken-or-the-egg problem. If somebody makes a effort to make an egg, that will eventually grows up to a chicken.
• Should I re-implement all the Vim script codes in my Vim plugin? That's too much work...
  • No. Running profiler and writing the bottleneck part in Lua is usually enough. Keep the pure Vim script part for people who don't have if_lua for a while.
• Which plugins are using if_lua for performance? I'd like to refer and learn the technique there.
  • neocomplete.vim this requires if_lua
  • unite.vim if_lua is optional just for speed
• What if_lua can't do whereas others like if_python can do?
  • Multithreading. This is pros.
  • Sockets. You may want to ask users to install vimproc to use network connections.
  • Non-blocking IO. Use vimproc instead like above.
  • In short: if_lua is like faster Vim script; it doesn't add superpower.

Past discussions

• Why not python interface?

Vim Advent Calendar 2012

There is a tech advent calendar ongoing since December 2012 which is all about Vim. This article is for that as its 266th day.

• Vim Advent Calendar 2012 http://atnd.org/events/33746

I previously posted an article for Vim Advent Calendar 2012 about if_lua and the patch described above on January 17, 2013. The post is http://vim-users.jp/2013/01/vim-advent-calendar-2012-ujihisa-2/ and is written in Japanese.

Day 4 -- JavaScript in a week

previous post <-> next post

I accidentally started playing SimCity3000 and it ate big amount of my time..

return

https://github.com/ujihisa/jinaw/commit/2dbc9ed0b3240d9081932c5a519a9767a3763ee4

return is the only one feature to cancel executing the rest of statements, since there's no try/catch exception handling in this subset. Since the run- function to sequentially execute statements isn't something like a loop but an explicit recursion, to implement return, just simply don't recur.

(run '[(var f (function []
                        [(fcall 'console.log [1])
                         (return 9)
                         (fcall 'console.log [2])]))
       (fcall 'console.log [(fcall 'f [])])])

This outputs 1 and 9 but not 2.

typeof

https://github.com/ujihisa/jinaw/commit/c499b1bc23a11799ffe119d8a974729c6936c4ad

JavaScript has a unary operator typeof to tell the type of a value/variable-name.

> typeof 1
"number"
> var x = 1
> typeof x
"number"
> typeof y
"undefined"

Since typeof isn't a function it has more power than functions; it also accepts undefined variables without throwing ReferenceError.

(run '[(fcall 'console.log [(typeof 1)]);=> "number"
       (var x 1)
       (fcall 'console.log [(typeof x)]);=> "number"
       (fcall 'console.log [(typeof y)]);=> "undefined"
       (fcall 'console.log [(typeof (fcall '+ ['x 2]))])]);=> "number"

==

== is one of the most difficult aspect of JavaScript. The behavior is too tricky to understand.

> 1 == ['1']
true
> [] == []
false
> [] == ''
true
> 1 == true
true
> 1 == 'true'
false
> '1' == true
true

According to specification, the definition of == is

If the two operands are not of the same type, JavaScript converts the operands then applies strict comparison. If either operand is a number or a boolean, the operands are converted to numbers if possible; else if either operand is a string, the other operand is converted to a string if possible. If both operands are objects, then JavaScript compares internal references which are equal when operands refer to the same object in memory.

Well, I couldn't clearly understand it. What if one operand is a number and the other is a boolean? What is an operand is a number but the other operand isn't possible to be converted? They aren't written.

I gave up and made one that works fine for now..

(run '[(fcall 'console.log [(fcall '== [1 1])]);=> true
       (fcall 'console.log [(fcall '== [1 2])]);=> false
       (fcall 'console.log [(fcall '== ["1" 1])]);=> true
       (fcall 'console.log [(fcall '== ["1" 2])]);=> false
       (fcall 'console.log [(fcall '== ["1" "1"])]);=> true
       (fcall 'console.log [(fcall '== ["1" "2"])]);=> false
       (fcall 'console.log [(fcall '== ["aaa" "aab"])])]);=> false

[]

https://github.com/ujihisa/jinaw/commit/8878e40f5a21f8a79067aa822d088db12cb87029

[] is a unary operator for objects. Note that "object" is a hash-map or an array in JavaScript terminology. I named it as aref borrowed from Ruby's parser which means array ref, since Clojure doesn't recognize '[] as Symbol but as vector. To distinguish it with user-defined something which name is aref, I named it as -aref.

(run '[(fcall 'console.log [(fcall '-aref [{"0" "a" "1" "b" "2" "c"} 0])])]);=> "a"
(run '[(fcall 'console.log [(fcall '-aref [{"0" "a" "1" "b" "2" "c"} "1"])])]);=> "b"
(run '[(fcall 'console.log [(fcall '-aref [{"0" "a" "1" "b" "2" "c"} 3])])]);=> "undefined"
(run '[(fcall 'console.log [(fcall '-aref [{"0" "a" "1" "b" "2" "c"} "4"])])]); => "undefined"

Day 3 -- JavaScript in a week

previous post <-> next post

I made many commits but most of them were improvements for internal structure. Actual new features are if and (strict) equal/notequal.

if

I implemented if as an expression like a ? b : c as below.

(defn evaluate [expr env]
  "assumption: env won't change"
  (if (list? expr)
    (let [[car & cdr] expr]
      (case car
        if (let [[cond- then- else-] cdr]
             (if (js-boolean (evaluate cond- env))
               (evaluate then- env)
               (evaluate else- env)))
        ...snip...

Too straightforward to explain. Since cond looked ambiguous to Clojure's one, so I added dash as suffix, and just for naming consistency, I added it to all the names..

equal/not equal

Strict equal (===)

Returns true if the operands are strictly equal (see above) with no type conversion.
https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Operators/Comparison_Operators

Luckily I could simply use Clojure's = function.

(defn evaluate [expr env]
  "assumption: env won't change"
  (if (list? expr)
    (let [[car & cdr] expr]
      (case car
        if (let [[cond- then- else-] cdr]
             (if (js-boolean (evaluate cond- env))
               (evaluate then- env)
               (evaluate else- env)))
        function (let [params (first cdr)
                       body (second cdr)]
                   {:type :function :params params :body body})
        fcall (let [func (evaluate (first cdr) env)
                    args (map #(evaluate % env) (second cdr))]
                (case func
                  console.log (println (js-string (first args)))
                  + (if (every? number? args)
                      (+ (first args) (second args))
                      (str (js-string (first args)) (js-string (second args))))
                  === (= (first args) (second args))
                  !== (not= (first args) (second args))
                  (if (= (:type func) :function)
                    (let [applied-params (into {} (map (fn [x y] [x y])
                                                       (:params func)
                                                       args))]
                      (run- (:body func) (merge env applied-params)))
                    (prn 'must-not-happen 'missing-function func))))
        quote (get env (first cdr) 'missing-local-var)
        expr))
    expr))

Now you have if and a predicate function, you can make a safe loop with recursive function calls.

var f = function(n) {
  (n === 10) ? console.log('end') : f(n + 1);
}
f();

It'll be represented as the below.

(run '[(var f (function [n]
                        [(fcall 'console.log ['n])
                         (if (fcall '=== ['n 10])
                           (fcall 'console.log ["end"])
                           (fcall 'f [(fcall '+ ['n 1])]))]))
       (fcall 'f [0])])

It outputs 0 to 10 and "end"

Refactoring to make it more declarative

You may have noticed that the evaluate function is already long and have guessed that it would get longer and longer as I add more builtin functions. First I made change to separate each built-in functions with updating a global variable *builtins*. That's still not well readable because of the boilerplate code. I made another change to provide defbuiltin macro. Now you can add a new builtin function with like the following code.

(defbuiltin + [x y]
  (if (and (number? x) (number? y))
    (+ x y)
    (str (js-string x) (js-string y))))

It looks very like Clojure's defn.

Day 2 -- JavaScript in a week

previous post <-> next post

summary:

• type conversions
  • JavaScript's Boolean(), Number() and String()
  • console.log and + now implicitly convert its arguemnts into String
• function literal without closure
  • (function(x) { console.log(x); })(2) works

Type conversions: Boolean(), Number() and String()

https://github.com/ujihisa/jinaw/commit/f28e7f55559f4919d6718539d413f0e65bd4a9e0

(defn js-boolean
  "https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Boolean"
  [value]
  (not (get #{0 'null false 'NaN 'undefined} value false)))

(defn js-number
  "https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Number"
  [value]
  (condp instance? value
    String (if (empty? value)
             0
             (let [x (read-string value)]
               (if (number? x) x 'NaN)))
    Long value
    'NaN))

(defn js-string
  "https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/String"
  [value]
  (.toString value))

Looks fairly straightforward.

Type conversions in + operator

It works both for numbers and strings.

• • for numbers https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Operators/Arithmetic_Operators
• • for strings https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Operators/String_Operators

The rule which to choose is really simple. If either one of the operands is not a number, it chooses the string's one and converts both operands to strings.

> 1 + 2
3
> '1' + 2
"12"
> 1 + '2'
"12"
> '1' + '2'
"12"
> [1] + [2]
"12"
> 1 + [2]
"12"

so

(defn evaluate [expr env]
  "assumption: env won't change"
  (if (list? expr)
    (let [[car & cdr] expr]
      (case car
        fcall (let [func (evaluate (first cdr) env)
                     args (map #(evaluate % env) (second cdr))]
                 (case func
                   console.log (println (js-string (first args)))
                   + (if (every? number? args)
                       (+ (first args) (second args))
                       (str (js-string (first args)) (js-string (second args))))
                   (prn 'must-not-happen 'missing-function func)))
        quote (get env (first cdr))
        expr))
    expr))

now the runtime outputs "1hello" by console.log(1 + 'hello').

function literal without closure

https://github.com/ujihisa/jinaw/commit/a50de0f344951e2e36a60131458d8533ea75241a

A function object which doesn't have lexical scope is represented simply as a tuple of parameter names and body (a sequence of statements.) Here just for readability in the future I'll use a hash-map which has :type, :params, and :body for function objects.

(defn evaluate [expr env]
  "assumption: env won't change"
  (if (list? expr)
    (let [[car & cdr] expr]
      (case car
        function (let [params (first cdr)
                       body (second cdr)]
                   {:type :function :params params :body body})
        fcall (let [func (evaluate (first cdr) env)
                    args (map #(evaluate % env) (second cdr))]
                (case func
                  console.log (println (js-string (first args)))
                  + (if (every? number? args)
                      (+ (first args) (second args))
                  (if (= (:type func) :function)
                    (let [applied-params (into {} (map (fn [x y] [x y])
                                                       (:params func)
                                                       args))]
                      (run- (:body func) (merge env applied-params)))
                    (prn 'must-not-happen 'missing-function func))))
        quote (get env (first cdr) 'missing-local-var)
  ...snip...

It doesn't support return yet.

Day 1 -- JavaScript in a week

https://github.com/ujihisa/jinaw

I'll make an implementation of a subset of programming language JavaScript within a week.

next post

Defining a subset of JavaScript

The implementation should execute the following JavaScript code.

var acc = 'dummy';
var f = function() {
  var acc = [];
  return function(x) {
    return x ? acc.push(x)&&f(x - 1) : acc;
  };
  console.log("this won't show up!");
}();

String.prototype.ujihisa = function() {
  var x = 10;
  return this + f(x);
}

console.log("hello".ujihisa());
console.log(acc == 'dummy');
console.log({'a': 2}.a == 2);

output:

hello10,9,8,7,6,5,4,3,2,1
true
true

That sample JS code covers the following important aspects.

• statements and expressions
• literals (String, Number, Function, Array and Object)
• function call, method call, operators, and index/key access of Object
• implicit convertion (String + Number)
• lexical scope (see acc var)
• method addition by prototype
• return exits from the current function

The implementation consists of the following 2 parts.

• parser (JS code to internal representation in S-expression)
• runtime

I'll work on runtime first in Clojure, then work on parser in Haskell with parsec.

parsing

I don't implement the parser for now but I simply parse the sample code by hand. the first 2 statements will be like the below.

before:

var acc = 'dummy';
var f = function() {
    var acc = [];
    return function(x) {
        return x ? acc.push(x)&&f(x - 1) : acc;
    };
}();

after:

'[(var 'acc "dummy")
  (var 'f
       (call
         (function []
                   [[(var 'acc (array))
                     (return
                       (function ['x]
                                 [[(return (if 'x
                                             (and (mcall 'push 'acc ['x])
                                                  (call 'f [(minus 'x 1)]))
                                             'acc))]]))]])
         []))]

Day 1 progress

https://github.com/ujihisa/jinaw/commit/6902cd4e9382e2da58fb9d43f8069256e8fe1ded

• just finished implementing var and console.log

  (run '[(var x "hello")
         (fcall 'console.log ['x])])

• currently console.log is just a single name of built-in function

https://github.com/ujihisa/jinaw/commit/30ad8c898a1c806735eaa990656258537b65a34c

• changed the structure a little bit.. to separate run to handle statements and evaluate to handle an expression

• now it supports nested function calls

  (run '[(var x 1)
         (fcall 'console.log [(fcall '+ ['x 2])])])

I'll be in Japan and Taiwan for a while

• Taiwan: Oct 28
• Japan: Oct 28 to Nov 21
• Taiwan: Nov 21 to Nov 23

I have no particular plans in Taiwan yet besides having awesome 小籠包s there.

小籠包 from wikipedia

Whereas I have plans in Japan -- I'll be organizing 4 conferences/meetups there.

• ujihisa.vim#3: Vim conference
• JVM study meetup: about Scala and Clojure
• Vim study meetup
• LiveCoding#11

If you are interested in either some of the topics above or me, and have plan to visit Kanto or Kansai area, you should check it out and give a presentation there.