Org Reflect: Mirror source code from files into Org Src blocks (Emacs package)
Below you find the latest version of (1) the package's README and (2) its main source file.
For the git repository and issues tracker, see the project's page on sr.ht.
For more packages, see Software.
I use this package often. In fact, this very page uses it.
The Contents of source are inserted with this:
#+include: /path/to/org-reflect.el src emacs-lisp
but the Structure, which comes right before it, is generated by this block:
#+begin: reflect :file /path/to/org-reflect.el :sh (sed -n "/^;;;\\+ /p") #+end:
A C-c C-c on it inserts an emacs-lisp src block inside it, whose contents are the fetched file's contents with a sed filter applied.
README.org
Overview
Org Reflect is an Emacs package that allows you to mirror the contents of source files into a target org file. When the contents are code, they are automatically wrapped in an Org Src block (this behavior can be disabled, if you want). Before being displayed, filters can be applied, such as restriction of line ranges and the processing of the text through Emacs Lisp or Shell script filters of your choice (write any functions you want).
Therefore, it embeds data from other files into your Org document. This is convenient, and can be instantly updated.
It’s particularly useful if you are producing documents that quote from other documents — so you don’t need to copy-paste. Moreover, contrary to the #+include tag from Org Export, you can visualize the results, which are actually copied to the target Org file itself as displayed.
Installation
See my page Software for the most up-to-date instructions on how to download and install any of my Emacs packages.
Having downloaded and installed the package and its dependencies, adapt the configurations below to your init.el file.
(use-package org-reflect :after org ;; Without this, C-c C-c will only work to update a reflect block after you ;; used org-update-all-dblocks at least once. If you remove the :demand, you ;; must "C-c u" in some org file to activate the function. :demand t ;; Optional (or another key of your choice): :bind ("C-c u" . org-update-all-dblocks))
Alternatively, if you don’t have use-package, add to your init:
(require 'org-reflect) (bind-key "C-c u" #'org-update-all-dblocks)
Finally, check the custom variables — org-reflect-org-structure-key in particular, so that after running M-x org-reflect-update-org-structure-template-alist you will be able to quickly insert org-reflect blocks.
Usage
An Org Reflect block can be easily inserted with a shortcut from org-structure-template-alist — that thing that also lets you insert Org Src blocks. This shortcut can be added either by running this command:
M-x org-reflect-update-org-structure-template-alist
or by manually adding this to your init.el:
'(org-structure-template-alist '(("r" "#+begin: reflect :file ?\n#+end:" "") ;; ... ;; other shortcuts ;; ... ))
or directly to the interface.
Once this has been done, a simple stroke of < r TAB on an empty line inserts this:
#+begin: reflect :file ٭ #+end:
with point after :file. From there, you specify what you want, and make it happen with C-c C-c.
What can be done is better shown through examples. Lots of examples.
Many examples
(These come straight from examples/example.org, which you can open and see as .org if you wish.)
* Examples of org-reflect usage ** Simple reflection, whole file, autodetection Our example data is a shell script with five lines of echo commands. #+begin: reflect :file example.sh #+begin_src shell echo 1 echo 2 echo 3 echo 4 echo 5 #+end_src #+end: ** Empty file We can omit :file and start from scratch: #+begin: reflect :sh (sed "s/^/Hello there!/") Hello there! #+end: ** Line restrictions #+begin: reflect :file example.sh :lmin 3 #+begin_src shell echo 3 echo 4 echo 5 #+end_src #+end: #+begin: reflect :file example.sh :lmax 3 #+begin_src shell echo 1 echo 2 echo 3 #+end_src #+end: #+begin: reflect :file example.sh :lmin 2 :lmax 3 #+begin_src shell echo 2 echo 3 #+end_src #+end: The :lr field has precedence and overwrites any :lmin and :lmax. #+begin: reflect :file example.sh :lmin 2 :lmax 3 :lr (1 4) #+begin_src shell echo 1 echo 2 echo 3 echo 4 #+end_src #+end: #+begin: reflect :file example.sh :lr (2 4) #+begin_src shell echo 2 echo 3 echo 4 #+end_src #+end: #+begin: reflect :file example.sh :lr (3) #+begin_src shell echo 3 echo 4 echo 5 #+end_src #+end: #+begin: reflect :file example.sh :lr (nil 3) #+begin_src shell echo 1 echo 2 echo 3 #+end_src #+end: #+begin: reflect :file example.sh :lr (2 2) #+begin_src shell echo 2 #+end_src #+end: ** With no wrapping into a src block — just plain text #+begin: reflect :file example.sh :lr (2 4) :blk no echo 2 echo 3 echo 4 #+end: ** Choosing a different type *** Another src type By extension: #+begin: reflect :file example.sh :lr (2 4) :type hs #+begin_src haskell echo 2 echo 3 echo 4 #+end_src #+end: By name: #+begin: reflect :file example.sh :lr (2 4) :type python #+begin_src python echo 2 echo 3 echo 4 #+end_src #+end: (No, it doesn't "make sense". But it's a simple example, easy to understand.) *** A special block #+begin: reflect :file example.sh :lr (2 4) :type example #+begin_example echo 2 echo 3 echo 4 #+end_example #+end: #+begin: reflect :file example.sh :lr (2 4) :type quote #+begin_quote echo 2 echo 3 echo 4 #+end_quote #+end: ** Pre-reflection filters *** The :sh pre-reflection filter — applying Shell before reflection **** Simple one #+begin: reflect :file example.sh :sh (tac | rev | sed "s/^/# /") #+begin_src shell # 5 ohce # 4 ohce # 3 ohce # 2 ohce # 1 ohce #+end_src #+end: **** From an empty file Making a code block from scratch by omitting :file and its value. #+begin: reflect :sh (sed "s/^/echo 'Hello world'/") :type bash #+begin_src bash echo 'Hello world' #+end_src #+end: **** Passing header options Using default expansions #+begin: reflect :file example.sh :lr (2 4) :args "$! !ube bash" #+begin_src shell :shebang #!/usr/bin/env bash echo 2 echo 3 echo 4 #+end_src #+end: Using special expansions #+begin: reflect :file example.sh :args $ro-p :sh (tac) #+begin_src shell :results output pp echo 5 echo 4 echo 3 echo 2 echo 1 #+end_src #+end: Quotation is needed when you want to pass multiple :args #+begin: reflect :file example.sh :lr (2 4) :args "$tn $rvv" :sh (tac) #+begin_src shell :tangle :results value verbatim echo 4 echo 3 echo 2 #+end_src #+end: You may mix :args expansions with literal ones: #+begin: reflect :file example.sh :lr (2 4) :args "$r--r :tangle no $s my-session" #+begin_src shell :results raw :tangle no :session my-session echo 2 echo 3 echo 4 #+end_src #+end: You can also use invalid or blank values (or just none)\\ to then pass your own parameters: #+begin: reflect :file example.sh :lr (2 4) :args "$t- ~/my/path" #+begin_src shell :tangle ~/my/path echo 2 echo 3 echo 4 #+end_src #+end: **** Complicating it #+begin: reflect :file example.sh :sh ((tac | rev && printf "\n%s" "24 ohce") | sed "/^3/ d" | rev) #+begin_src shell echo 5 echo 4 echo 2 echo 1 echo 42 #+end_src #+end: **** Anaphoric workaround: reading the whole string and assigning it to a variable The || is used instead of a semicolon, because semicolons wouldn't work here\\ (they're not parsed, not even if escaped). The read command never finds the\\ null byte while reading the whole string, so it returns 1 (as if an error).\\ This is why the recipe works. #+begin: reflect :file example.sh :lr (2 4) :sh (read -rd\0 it || printf "%s\n\n%s" "$it" "$it") #+begin_src shell echo 2 echo 3 echo 4 echo 2 echo 3 echo 4 #+end_src #+end: **** Another anaphoric one, with changed type Prints a range twice — once with characters reversed by line, once with lines\\ reversed. And it isn't a shell src block any more, because the explicit :type overwrites\\ filetype autodetection by file extension. - Note 1 :: The <<< thing is just the shamefully underused Here String, not\\ some extra feature of org-reflect. If you've never seen it, run man bash,\\ then /<<< for info. - Note 2 :: Examples here are Bash. If your shell-file-name points to Zsh\\ because that's your default, then use Zsh syntax. #+begin: reflect :file example.sh :lr (2 4) :sh (read -rd\0 it || rev <<< "$it" && tac <<< "$it") :type example #+begin_example 2 ohce 3 ohce 4 ohce echo 4 echo 3 echo 2 #+end_example #+end: **** Escaping code that could be confused as org ***** An org block: the asterisk and the hashtag are comma-escaped, as expected If you C-c ' inside the block, it's org. #+begin: reflect :file example.sh :lr (3) :sh (sed "1i** This is a tree\\n#+property: vegetal") :type org #+begin_src org ,** This is a tree ,#+property: vegetal echo 3 echo 4 echo 5 #+end_src #+end: **** Passing script files as argument Here, the filter.sh file contains a single command: rev. So it receives a\\ string, piped to it. But you can make that script as complex as you need. #+begin: reflect :file example.sh :sh (./filter.sh | sed "s/^/# /") :type py #+begin_src python # 1 ohce # 2 ohce # 3 ohce # 4 ohce # 5 ohce #+end_src #+end: Or: #+begin: reflect :file example.sh :sh (read -rd\0 it || ./filter.sh <<< "$it" | sed "s/^/# /") :type py #+begin_src python # 1 ohce # 2 ohce # 3 ohce # 4 ohce # 5 ohce #+end_src #+end: Or: #+begin: reflect :file example.sh :sh (read -rd\0 it || echo "$it" | ./filter.sh | sed "s/^/# /") :type py #+begin_src python # 1 ohce # 2 ohce # 3 ohce # 4 ohce # 5 ohce #+end_src #+end: Since: 1. unquoted ; is parsed as comment by Elisp, and 2. the read command necessarily returns 1, because while reading the whole\\ string (what we want it to do) it never finds the null character,\\ we use || above instead. The recipe works.\\ *** The :el pre-reflection filter — applying Emacs Lisp before reflection While in Bash, as the previous examples show, we use read -rd\0 it,\\ here in Elisp our contents are always under the let-bound variable s. So: #+begin: reflect :file example.sh :el (upcase s) :blk no ECHO 1 ECHO 2 ECHO 3 ECHO 4 ECHO 5 #+end: #+begin: reflect :file example.sh :lr (2 4) :el (replace-regexp-in-string "echo" "Say" s) :type example #+begin_example Say 2 Say 3 Say 4 #+end_example #+end: Using the dash and s libraries: #+begin: reflect :file example.sh :el (->> s capitalize (s-replace-all '(("4" . "4\n") ("5" . "no more!")))) :type verse #+begin_verse Echo 1 Echo 2 Echo 3 Echo 4 Echo no more! #+end_verse #+end: #+begin: reflect :file example.sh :el (->> s s-lines (remove "") (-map 'split-string) (--map `(,(read (cadr it)) . ,(car it))) (format "%S")) :type el #+begin_src emacs-lisp ((1 . "echo") (2 . "echo") (3 . "echo") (4 . "echo") (5 . "echo")) #+end_src #+end: *** Chaining: applying more than one reflection filter Current status: 1. Only at most one of each type is allowed. 2. Filters are applied in the order in which they appear. 3. Available filters: Shell and Elisp. **** What happens if you apply two of the same kind of filter? Only the first one will be used. #+begin: reflect :file example.sh :sh (rev) :sh (tac) :sh (sed 2d) #+begin_src shell 1 ohce 2 ohce 3 ohce 4 ohce 5 ohce #+end_src #+end: So instead of chaining :sh filters, chain pipes inside the same filter: #+begin: reflect :file example.sh :sh (rev | tac | sed 2d) #+begin_src shell 5 ohce 3 ohce 2 ohce 1 ohce #+end_src #+end: **** Applying an elisp filter, and then a shell filter #+begin: reflect :file example.sh :el (capitalize s) :sh (rev) #+begin_src shell 1 ohcE 2 ohcE 3 ohcE 4 ohcE 5 ohcE #+end_src #+end: **** Applying a shell filter, and then an elisp filter #+begin: reflect :file example.sh :sh (rev) :el (capitalize s) #+begin_src shell 1 Ohce 2 Ohce 3 Ohce 4 Ohce 5 Ohce #+end_src #+end: #+begin: reflect :file example.sh :lr (2 2) :sh (sed "s/^/> /;s/$/ -- The answer/") :el (s-replace "echo" "40 +" s) :type lhs #+begin_src haskell-literate > 40 + 2 -- The answer #+end_src #+end: * Did you find an interesting use case and/or example? I would love to hear it!
Another example
To dogfood my own package, I’ll add here an org-reflect block that fetches the first four lines of the main docstring of the package and transforms it for display in this README.org file, for later export as html, which is how you’re probably reading it.
Here’s how it ended up:
#+begin: reflect :file org-reflect.el :sh (sed -n "/defun org-dblock-write:reflect/,$ p" | sed -n "2,5p") :el (concat (s-trim-right s) "\"") #+end:
Result:
"Reflect the contents of a file according to PARAMS. Fetch file contents and insert in a dynamic org block. A \\[org-ctrl-c-ctrl-c] on the block’s header updates the displayed contents."
The current list of special expansions
(org-reflect-show-special-expansions) (with-current-buffer "*Org Reflect Special Expansions*" (let ((bufstr (buffer-string))) (f-write bufstr 'utf-8 "special-expansions.el")))
Org Reflect Special Expansions ============================== (see also those in ‘org-reflect-args-expansions-default’)
Examples: :args "$C0 $R1" ==> :colnames no :rownames yes :args "$t ~/o.el" ==> :tangle ~/o.el :args $rvv ==> :results value verbatim :args $r--c ==> :results code :args $r-l-a ==> :results list append :args $r---s ==> :results silent :args $xb ==> :export both
Special Expansions: :r :results (:o output :v value) (:f file :l list :t table :v verbatim) (:c code :d drawer :h html :l latex :o org :p pp :r raw) (:a append :n none :p prepend :r replace :s silent) :n :noweb (:0 no :1 yes :e eval :t tangle :n no-export :s strip-export) :C :colnames (:0 no :1 yes :n nil) :R :rownames (:0 no :1 yes) :a :cache (:0 no :1 yes) :c :comments (:0 no :1 yes :b both :l link :o org :n noweb) :e :eval (:0 no :q query :n never-export :x query-export) :h :hlines (:0 no :1 yes) :m :mkdirp (:0 no :1 yes) :p :padline (:0 no :1 yes) :t :tangle (:0 no :1 yes) :x :export (:b both :c code :n none :r results) :! :shebang :P :prologue :E :epilogue :N :noweb-ref :S :noweb-sep :T :tangle-mode :f :file :d :file-desc :s :session :u :no-expand :w :wrap Type those (no abbreviations): :dir :sep :var :post
Features or limitations?
The visualized results:
- are exclusively unidirectional: from the file to the Org block, and
- need to be manually refreshed (not on-the-fly updates).
Depending on what you want, this is either a feature or a limitation.
- If you want the convenience of editing source files from either the source or the Org block, then Org Reflect can't help you. Anything you manually edit on the source block will be undone by the next C-c C-c refresh and won't update the source file. But this also means that your source file is preserved, and the source block can be seen as the result of a function (which could be #'identity) applied on it.
- Not on-the-fly means you can't have some sort of passive continuous monitoring of the contents of a file as it changes. But it also means you get to preserve a snapshot of the source file on your Org document, and therefore have control over if and when it is updated.
Is bidirectionality desirable, though? Would it be good to have the option to manually "reflect back" to the original file the contents of the block after you manually changed it? Maybe it would, and this could be a feature for a future release.
Security considerations
Org-reflect uses dynamic blocks, which allow for arbitrary code execution. So you shouldn’t execute this:
#+begin: reflect :file somefile :sh (evil-command) :el (another-evil-command) #+end:
The execution of such blocks usually happens just like the execution of regular Org Src blocks: through C-c C-c. This means you’d have to press these keys to execute it.
For regular Org Src blocks (including the ones that Org Reflect create to wrap around reflected code), you can set the variable org-confirm-babel-evaluate to t ot nil depending on whether you want confirmation before every execution. This doesn’t seem to be possible with dynamic blocks. If you C-c C-c on it, it will execute.
Assuming that org-reflect’s code itself is safe, the risk should be limited to the presence and contents of any :sh and :el values in the header. In my estimation, in the absence of critical bugs that I might have overlooked, if there are no :sh or :el headers, or if you understand the commands they’re calling, it’s probably ok.
“Probably” because they might still exist. If you see any, then please report it. I’d like security risks to be limited to the risks implied by whatever code is executed in these headers (which users will write at their own discretion) — and nothing else: there should be no security risks from org-reflect’s code itself. I currently see none, and have been using this package myself. Still, your machine is a different one, and GPL’s “no guarantees” warnings apply. Use with care, scrutinize the code if you wish, and please report bugs.
Now a note about automation. You can manually use this command to update all dblocks in an org file, including org-reflect ones:
M-x org-reflect-update-all-dblocks
and you could bind a key to it.
This is handy, but care is likewise recommended: use it only if you trust all dynamic blocks in the file, org-reflect ones included. This is like applying C-c C-c to all of them. You need to trust them all.
Finally, at the next level of automation, to have blocks auto-update when you save the org files they are in, you could add this to your init file:
;; CAREFUL! See warnings below. (use-package org ;; ... ;; other org configs here ;; ... :hook (org-mode . (lambda () (add-hook 'before-save-hook #'org-reflect-update-all-dblocks))))
Or if you don’t use use-package, simply put this in your init.el:
;; CAREFUL! See warnings below. (add-hook 'before-save-hook #'org-reflect-update-all-dblocks)
because org-update-all-dblocks does nothing outside org mode (and derived), so a general before-save-hook should be enough.
Now, I DO NOT particularly recommend that you do this, unless you know well what you’re doing. Why? Because it exposes you to autoloading blocks from unknown org files that you download from the internet. You might prefer to be deliberate about running code in org files — something that prompts your action instead of it being automatic.
Contributing
See my page Software for information about how to contribute to any of my Emacs packages.
License
This project follows the REUSE Specification (FAQ), which in turn is built upon SPDX.
Therefore, license and copyright information can be found in:
- each file's comment header, or
- an adjacent file of the same name with the additional extension
.license, or - the
.reuse/dep5file
The full text of the licenses can be found in the LICENSES subdirectory.
org-reflect.el
Structure
;;; org-reflect.el --- Mirror source code from files into Org Src blocks -*- lexical-binding: t -*- ;;; Commentary: ;;;; For all the details, please do see the README ;;; Acknowledgments: ;;; Code: ;;;; Libraries ;;;; Backports ;;;; Package metadata ;;;; Customizable variables ;;;; Functions ;;;;; Utilities ;;;;; Core ;;;; Wrapping up ;;; org-reflect.el ends here
Contents
;;; org-reflect.el --- Mirror source code from files into Org Src blocks -*- lexical-binding: t -*- ;; SPDX-FileCopyrightText: © flandrew <https://keyoxide.org/191F5B3E212EF1E515C19918AF32B9A5C1CCCB2D> ;; SPDX-License-Identifier: GPL-3.0-or-later ;;--------------------------------------------------------------------------- ;; Author: flandrew ;; Created: 2021-06-04 ;; Version: 0.5.7-git ;; Homepage: <https://flandrew.srht.site/listful/software.html> ;; Keywords: outlines, tools, convenience ;; Package-Requires: ((emacs "25.1") (dash "2.12") (s "1.7")) ;;--------------------------------------------------------------------------- ;; This file is part of Org Reflect, which is NOT part of GNU Emacs. ;; This program is free software: you can redistribute it and/or modify it ;; under the terms of the GNU General Public License as published by the Free ;; Software Foundation, either version 3 of the License, or (at your option) ;; any later version. ;; ;; This program is distributed in the hope that it will be useful, but WITHOUT ;; ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or ;; FITNESS FOR A PARTICULAR PURPOSE. For more details, see the full license at ;; either LICENSES/GPL-3.0-or-later.txt or <https://www.gnu.org/licenses/>. ;;; Commentary: ;; ;; Org Reflect allows you to mirror the contents of source files into a target ;; org file. When the contents are code, they are automatically wrapped in an ;; Org Src block (this behavior can be disabled, if you want). Before being ;; displayed, filters can be applied, such as restriction of line ranges and ;; the processing of the text through Emacs Lisp or Shell script filters of ;; your choice (write any functions you want). ;; ;; Therefore, it embeds data from other files into your Org document. This is ;; convenient, and can be instantly updated. ;; ;; It’s particularly useful if you are producing documents that quote from ;; other documents — so you don’t need to copy-paste. Moreover, contrary to ;; the #+include tag from Org Export, you can visualize the results, which are ;; actually copied to the target Org file itself as displayed. ;; ;;;; For all the details, please do see the README ;; ;; Open it easily with any of the below: ;; (find-file-read-only "README.org") <--- C-x C-e here¹, or ;; (find-file-read-only-other-frame "README.org") <--- C-x C-e here¹, or ;; (find-file-read-only-other-window "README.org") <--- C-x C-e here¹ ;; ;; or read it online: ;; <https://flandrew.srht.site/listful/sw-emacs-org-reflect.html> ;; ;; ¹ or the key that ‘eval-last-sexp’ is bound to, if not C-x C-e. ;; ;; (See also the docstring of ‘org-reflect-write’.) ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Acknowledgments: ;; ;; The tables in the docstring of ‘org-reflect-args-expansions-special’ were ;; constructed after studying: ;; - Org Manual’s (info "(org)Results of Evaluation") ;; - Org Babel Reference Card <https://github.com/fniessen/refcard-org-babel> ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Code: ;;;; Libraries (require 's) (require 'rx) (require 'dash) (require 'subr-x) ; ‘if-let’ (require 'org) ; ‘org-update-all-dblocks’ (require 'ob-core) ; ‘org-babel-chomp’ (require 'org-src) ; ‘org-escape-code-in-region’ (require 'lisp-mnt) ; ‘lm-summary’, ‘lm-version’, ‘lm-homepage’
;;;; Backports (eval-and-compile (defalias 'org-reflect--unbracket-string ;; Introduced somewhen after org 8.2.10 and before or at 9.0.3 (if (fboundp 'org-unbracket-string) 'org-unbracket-string (lambda (pre post string) (if (and (string-prefix-p pre string) (string-suffix-p post string)) (substring string (length pre) (- (length post))) string)))))
;;;; Package metadata (defconst org-reflect--name "Org Reflect") (defconst org-reflect--dot-el (format "%s.el" (file-name-sans-extension (eval-and-compile (or load-file-name buffer-file-name))))) (defconst org-reflect--readme-org (expand-file-name "README.org" (file-name-directory org-reflect--dot-el))) (defconst org-reflect--summary (lm-summary org-reflect--dot-el)) (defconst org-reflect--version (lm-version org-reflect--dot-el)) (defconst org-reflect--homepage (lm-homepage org-reflect--dot-el))
;;;; Customizable variables (defgroup org-reflect nil (format "%s." org-reflect--summary) :group 'org :link '(emacs-library-link :tag "Lisp file" "org-reflect.el") :link `(file-link :tag "README.org" ,org-reflect--readme-org) :link `(url-link :tag "Homepage" ,org-reflect--homepage)) (defcustom org-reflect-upcase-keywords-p nil "Whether you want to keep header keywords upcased or not. This package defaults to lowercase. So you’ll see blocks like this: #+begin: reflect :file ~/code/42.el #+begin_src emacs-lisp (+ 40 2) #+end_src #+end: whereas if you choose to upcase you’ll see: #+BEGIN: reflect :file ~/code/42.el #+BEGIN_SRC emacs-lisp (+ 40 2) #+END_SRC #+END:" :type 'boolean :group 'org-reflect) (defcustom org-reflect-org-structure-key (car (-difference (list "r" "R" "f" "F" "c" "C" "t" "T" "l" "L" "e" "E") (-map #'car org-structure-template-alist))) "Abbreviation key for structure completion. This is a string of usually one character (but can be more) to use as expansion with ‘org-structure-template-alist’. Type ‘<keyTAB’ and it expands into a new ‘reflect’ block. Make sure to run ‘org-reflect-update-org-structure-template-alist’ once to update it." :type 'string :group 'org-reflect) (defcustom org-reflect-org-structure-value "#+begin: reflect :file ?\n#+end:" "Abbreviation value for structure completion. This is the expansion of ‘org-reflect-org-structure-key’. The pair is added to ‘org-structure-template-alist’. Type ‘<keyTAB’ and it expands into a new ‘reflect’ block." :type 'string :group 'org-reflect) (defconst org-reflect-langs-default '(;; Shells (ash . ash) (bash . bash) (csh . csh) (dash . dash) (ksh . ksh) (mksh . mksh) (posh . posh) (shell . shell) ;; Other (awk . awk) (c . C) (clj . clojure) (conf . conf) (cpp . C++) (css . css) (dot . dot) (el . emacs-lisp) (hs . haskell) (html . html) (js . js) (l . picolisp) (lhs . haskell-literate) (lisp . lisp) (max . maxima) (md . markdown) (ml . ocaml) (org . org) (pl . perl) (py . python) (rb . ruby) (rkt . racket) (rs . rust) (scm . scheme)) "An association list of languages. Each pair represents: ‘(file-extension . org-babel-src name)’.") (defcustom org-reflect-langs-user nil "An alist of additional languages added by the user. Each pair represents: ‘(file-extension . org-babel-src name)’. You may add new languages according to what you use. This list will be checked before ‘org-reflect-langs-default’. A note about shell scripts. If your file has extension ‘bash’, it will create a src block ‘bash’. If, however, it has extension ‘sh’, org-reflect will look at the shebang on the first line and try to guess it. If, say, it finds a ‘#!/bin/bash’, it will use Bash. Etc. This is better, because it’s rare for people to make explicit in the extension the shell used. If the shebang says ‘#!/bin/sh’, it will default to just sh. If there is no shebang, it will default to ‘shell’. The logic here is that we’ll try to be as specific as possible about the shell language being used. If you want to override this behavior for some reason, just add one of these pairs to this variable here for the default behavior of what org-reflect will do if your file has a .sh extension: 1. ‘(sh . shell)’ will create a generic “Shell Script” block that will show up as ‘shell’. Note that if you share your org file, it will run the script under whatever default shell the user’s Emacs has specified under the variable ‘shell-file-name’. Check: (describe-variable 'shell-file-name). 2. ‘(sh . bash)’ will create a “bash” block that will show up as ‘bash’. Note that if your .sh file is pointing to some other shell, including plain sh (that is, usually dash), the org src block’s code may misrepresent the intended shell it was written on. 3. ‘(sh . sh)’ will create a “shell” block that will show up as ‘sh’. Note that if your .sh file is pointing to some other shell (such as bash, from its shebang), the org src block’s code may misrepresent the intended shell it was written on. If you think all this is a bit confusing, it’s because it actually is. If in doubt, leave the default. Unless it doesn’t work. Then try one of the above. You can test if execution is working under the hood by trying to \\[org-ctrl-c-ctrl-c] these block: #+begin_src bash [[ 4 > 2 ]] && rev <<< string #+end_src #+begin_src shell [[ 4 > 2 ]] && rev <<< string #+end_src #+begin_src sh [[ 4 > 2 ]] && rev <<< string #+end_src - If the first one works, then you have Bash. - If the second one works, then your default shell is Bash. - The third one shouldn’t work. That’s normal. Unless your /bin/sh is pointing to Bash instead of Dash, which would be unusual. Check with: ls -l /bin/sh. If you want to use Bash or other shell that isn’t what /bin/sh points to, add this pair to the variable ‘org-babel-load-languages’: (shell . t). For more on Org Babel’s treatment of shell languages, see the variable ‘org-babel-shell-names’ under ob-shell.el. For more on Org languages in general, check Info anchor ‘(org)Languages’ (Working with source code > Languages)." :type '(alist :value-type symbol) :group 'org-reflect) (defconst org-reflect-args-expansions-default '(("!ube" . "#!/usr/bin/env") ("!b/" . "#!/bin/")) "An alist of expansions that may be passed to :args. Each pair represents: ‘(abbreviation . expansion)’. They will be inserted in the org src block (or special block such as “verbatim” and “example”, when that is the case). You may add new ones by customizing ‘org-reflect-args-expansions-user’.") (defcustom org-reflect-args-expansions-user nil "An alist of user-added expansions that may be passed to :args. Each pair represents: ‘(abbreviation . expansion)’. You may add new ones according to what you normally use. This list will be checked before ‘org-reflect-args-expansions-default’. Prefixing the abbreviation with something unique is recommended, to avoid spurious expansions on other parts of the header by the find-replacement function. If you use a ‘$’ to prefix it, check if it isn’t already used by ‘org-reflect-args-expansions-special’, to avoid confusion. Note that ‘org-reflect-args-expansions-special’ is checked *before* this one, so your replacements will run on whatever is expanded by special." :type '(alist :key-type string :value-type string) :group 'org-reflect) (defconst org-reflect-args-expansions-special ;; Notes: ;; 1. No expansions for: :dir :sep :var :post ;; 2. The repetition of keys in the values’ cars below simplifies the code. '(;; With named values :r ((:r :results) ;; Collection, Type, Format, Handling (:o output :v value) (:f file :l list :t table :v verbatim) (:c code :d drawer :h html :l latex :o org :p pp :r raw) (:a append :n none :p prepend :r replace :s silent)) :n ((:n :noweb) (:0 no :1 yes :e eval :t tangle :n no-export :s strip-export)) :C ((:C :colnames) (:0 no :1 yes :n nil)) :R ((:R :rownames) (:0 no :1 yes)) :a ((:a :cache) (:0 no :1 yes)) :c ((:c :comments) (:0 no :1 yes :b both :l link :o org :n noweb)) :e ((:e :eval) (:0 no :q query :n never-export :x query-export)) :h ((:h :hlines) (:0 no :1 yes)) :m ((:m :mkdirp) (:0 no :1 yes)) :p ((:p :padline) (:0 no :1 yes)) :t ((:t :tangle) (:0 no :1 yes)) :x ((:x :export) (:b both :c code :n none :r results)) ;; Without named values :! ((:! :shebang)) :P ((:P :prologue)) :E ((:E :epilogue)) :N ((:N :noweb-ref)) :S ((:S :noweb-sep)) :T ((:T :tangle-mode)) :f ((:f :file)) :d ((:d :file-desc)) :s ((:s :session)) :u ((:u :no-expand)) :w ((:w :wrap))) "Expansions that may be passed to :args. This will be used before ‘org-reflect-args-expansions-user’ and ‘org-reflect-args-expansions-default’. Many header options have been coded for your convenience. In fact, almost all of them. In particular, currently, *ALL* possible combinations of ‘:results’, whether they make sense or not, may be expanded through nice, short variables. This will save you precious keystrokes, which will save you an untold amount of time, probably in the order of... many minutes per year, even! Here are the options for ‘:results’, and the corresponding values that Org Babel defaults to when they are not specified: | Classes | Options | |------------+-------------------------------------------------| | Collection | output value | | Type | file list table verbatim | | Format | code drawer html latex org pp raw | | Handling | append none prepend replace silent | | Classes | Default | Except when | which default | |------------+---------+-----------------------+---------------| | Collection | value | src: ledger | to output | | Type | table | res: drawer, org, raw | to verbatim | | Handling | replace | src: org, screen | to silent | \(Another exception: in code blocks of graphics-only languages, interpretation defaults to “file”.) To insert a ‘:results’ option combination, use a ‘$r’ followed immediately by the initial of each of the four desired classes, in the order above. To leave some of them unspecified, use a ‘_’ or a ‘-’ (as you prefer). So for example: $rvv__ => :results value verbatim $ro_r_ => :results output raw $r--c- => :results code $r---s => :results silent Unspecified options at the end may just as well be omitted: $rvv => :results value verbatim $ro_r => :results output raw $r--c => :results code The same applies to ‘:tangle’: a ‘$tn’ for “no”, a ‘$ty’ for “yes”. To use these both, then, add to org-reflect’s header: :args \"$rvv $tn\" Quotes may be omitted when there’s only one arg: :args $rvv To see them all, run the command ‘org-reflect-show-special-expansions’. For more, see Info anchor ‘(org)Results of Evaluation’ and Info anchor ‘(org)Working with source code’.") (defconst org-reflect-keywords '("#+begin" "#+end" "src") "Fixed words that will appear in org-reflect, src, and special blocks.") (defvar org-reflect-var-directory (if (boundp 'no-littering-var-directory) no-littering-var-directory (expand-file-name (convert-standard-filename "var/") user-emacs-directory)) "Default directory under which org-reflect subdir may be created.") (defcustom org-reflect-directory (expand-file-name (convert-standard-filename (concat org-reflect-var-directory "org-reflect/"))) "Where we store any data that org-reflect may eventually create. This suggested directory uses the ‘no-littering-var-directory’ default in order to try to keep your ‘user-emacs-directory’ clean. Unless you have good reasons, better not to change it." :type 'directory :group 'org-reflect) (defconst org-reflect-emptyfile-path (expand-file-name (convert-standard-filename "empty") org-reflect-directory) "An empty file. We use it whenever the value of ‘:file’ is nil. This also allows you to hack on top of empty org-reflect blocks, for any reason. For example, for quick creation of src blocks with specific headers and short filters.")
;;;; Functions ;;;;; Utilities (defun org-reflect--ensure-string (obj) "If OBJ is nil, return nil. If OBJ is a string, return it. If OBJ is a symbol or number, stringify it." (when obj (prin1-to-string obj 'no-escape))) (defun org-reflect--create-file (file &optional data) "If FILE doesn’t exist, touch it — that is, create it empty. If optional parameter DATA is passed, insert that. It inserts appending, but just as a precaution, since nothing should be done if file already exists." (unless (file-exists-p file) (make-directory (file-name-directory file) 'parents) (write-region (or data "") nil file 'append 'silent))) (defun org-reflect--filetype-from-shebang (file) "Given a FILE, try to get the name of the executable from its shebang." (when (file-exists-p file) (with-temp-buffer (insert-file-contents file) (--> (buffer-substring 1 (point-at-eol 1)) (when (s-match "^#!" it) (if (s-match (rx "/env" space) it) (-some->> it (s-match (rx "/env" (1+ space) (1+ (not (in space))))) car (s-chop-prefix "/env") s-trim) (-some->> it (s-match (rx (1+ (not (in "/" "!" space))) (or eos space))) car s-trim))))))) (defun org-reflect--normalize-trails (string &optional notrail) "Deal with trailing newlines in STRING. If NOTRAIL is non-nil, do not insert a trailing '\n'" (if string (concat (org-babel-chomp string) (unless notrail "\n")) (error "‘org-reflect--normalize-trails’: not a string"))) (defun org-reflect--header-lookup (arg &optional table) "Given an ARG (string) and a TABLE (a plist), return a string. If no TABLE is provided, default to ‘org-reflect-args-expansions-special’. If ARG is nil, return nil. ARG is a string such as “$rvv__”. The result from this would be the string “:results value verbatim”." (when arg (if (not (string= "$" (substring arg 0 1))) arg (let* ((table (or table org-reflect-args-expansions-special)) (keys (->> (split-string arg "" t "[$]") (--map (read (concat ":" it))))) (h-arg (car keys)) ;<--- to be kept hidden from teenagers. (klist (plist-get table h-arg)) (mapped (--map-indexed (plist-get (nth it-index klist) it) keys))) (->> mapped -non-nil (-map #'symbol-name) (s-join " "))))))
;;;;; Core ;;;###autoload (defalias 'org-reflect-write #'org-dblock-write:reflect) ;;;###autoload (defun org-dblock-write:reflect (params) "Reflect the contents of a file according to PARAMS. Fetch file contents and insert in a dynamic org block. A \\[org-ctrl-c-ctrl-c] on the block’s header updates the displayed contents. It tries to autodetect filetype, and when successful, wraps org src blocks around it. You can also specify it yourself. See ‘:type’ below. PARAMS are as follows (file is the only non-optional one): - ‘:file’ filename to transclude from. - ‘:lmin’ first line to fetch from this file (default: 1). - ‘:lmax’ last line to fetch from this file (default: file’s last). So without ‘:lmin’ and ‘:lmax’, the whole file is fetched. - ‘:lr’ lines range. If present, it overrides any ‘:lmin’ and ‘:lmax’ also present. Usage: :lr (2 5) ; lines 2–5 :lr (2) ; lines 2–end :lr (2 nil) ; lines 2–end :lr (nil 5) ; lines beg–5 :lr () ; all lines: same as omitting it altogether :lr (nil nil) ; all lines: same as omitting it altogether - ‘:type’ filetype as param for org src block to be included (default: inferred from file’s extension). If the type you entered is not in the ‘langs’ alist, it will make a special block. So if you enter ‘:type example’, it inserts a ‘#+begin_example’ block. But if you enter ‘:type el’ or ‘:type py’, it inserts ‘#+begin_src emacs-lisp’ or ‘#+begin_src python’ blocks, respectively. - ‘:blk’ boolean: whether to wrap contents inside an ‘org_src’ or ‘org_something’ block (default: t when ext is non-nil and match a language, or when you explicitly insert a ‘:type’; in other words, block is inserted by default). If you don’t want it inserted, add ‘:blk no’. (Note well: ‘no’, not nil.) - ‘:args’ any additional parameters to pass to the org_src block’s #+begin line if, and only if, a block is inserted. For #example, you may pass a ‘-n’ argument so that line #numbers are inserted when exported (note that in this #case the counting starts at 1, as usual, not at the #line number from the source file). Many predefined exist. They are handy and can save typing. For example: ‘$rvv’ => :results value verbatim So if you pass ‘:file ./this.hs :args $rvv’, your reflection would be wrapped up like this: #+begin_src haskell :results value verbatim (contents of ./this.hs file shown here) #+end_src To see them, run ‘org-reflect-show-special-expansions’ with \\[execute-extended-command]. You can add more by customizing ‘org-reflect-args-expansions-user’. Pre-reflection filters: - ‘:el’ elisp code to run over the fetched text before reflecting it. The contents of the text extracted from the indicated file will be assigned to the let-bound variable ‘s’. Between parentheses, add any elisp commands that you’d like to execute over this text. E.g.: :el (downcase s) would downcase all the text. And these are equivalent: :el ((lambda (x) (concat \"Example: \" (downcase x))) s) :el (concat \"Example: \" (downcase s)) :el (thread-last s downcase (concat \"Example: \")) :el (->> s downcase (concat \"Example: \")) ; with dash library. - ‘:sh’ shell code to run over the fetched text before reflecting it. The contents of the text extracted from the indicated file will be piped to the code. So between parentheses, add any shell commands that you’d like to execute over this text: :sh (tac | rev) would reverse the lines of the text and the text on each line; :sh (sed \"/TODO/ s/^/# /\") would comment every line containing the string ‘TODO’. Some caveats: 1. I couldn’t figure out how to pass a semicolon to it, because Elisp interprets it as comment. Consider using ‘&&’ instead if this is equally valid in your case. 2. If you want to capture it as a variable because of positional parameters, the usual shell tricks apply. For example, this: :sh (read -rd\\0 it || printf \"%s\" \"$it$it\") would read the whole string and print it twice. Why the ‘||’ here, you ask? Because the exit status will be 1 (error), since ‘read’ will reach the end of the string without finding a null character. But that’s what you want — to read the whole string. (Usually we’d separate it with a ‘;’ here.) 3. Splitting the command over multiple lines seems not to be possible. You’ll need to stay in the ‘#+begin:’ line. 4. For more complex stuff, consider packing your filter in a script that receives piped strings as input. Then run org-reflect with: :sh (~/bin/somescript.sh) or :sh (./somescript.sh) if it’s in the same folder as your org file and ‘M-x pwd’ matches that, or even :sh (./somescript.py) if you prefer other languages, or :sh (read -rd\\0 it || ./somescript.py \"$it\") to pass it as argument instead of piping to it. If you pick both, then they will be applied in the order given. If you repeat the same type of filter (e.g. two ‘:sh’), only the first is executed. Support for complex chaining is not planned. It can most likely be solved with a shell script that does all the chaining and is called as in #4 above. Nevertheless, extending filters to other languages is possible and shouldn’t be hard. If you’d like that, please see :sh’s syntax and propose an adaptation for your favorite language. Ideally, send the docstring and the elisp code. It’s worth emphasizing that the code of pre-reflection filters does *not* change the contents of the original file: it only changes the contents displayed." (-let [(beg end src) (--> org-reflect-keywords (if org-reflect-upcase-keywords-p (-map #'upcase it) it))] (let* ((file (--> (org-reflect--ensure-string (plist-get params :file)) (cond ((null it) (progn (org-reflect--create-file org-reflect-emptyfile-path) org-reflect-emptyfile-path)) ((file-exists-p it) it) (t (user-error "File %s doesn’t seem to exist" it))))) (ext (file-name-extension file)) (langs (append org-reflect-langs-user org-reflect-langs-default)) (args-exp (append org-reflect-args-expansions-user org-reflect-args-expansions-default)) (args (-some--> (org-reflect--ensure-string (plist-get params :args)) (split-string it) (-map #'org-reflect--header-lookup it) (s-join " " it) (if args-exp (s-replace-all args-exp it) it))) ;; Block types: ;; ‘type’ is what you entered after :type, if anything. ;; ‘type-t’ is the lookup of that on the ext→lang table, and when ;; not found, it then looks up the lang directly; so ;; you may enter either ‘:type’ hs or ‘:type’ haskell. ;; ‘type-x’ is the lookup of file extension on the ext→lang table. ;; and it will only happen if type-t is nil. ;; ‘type-s’ is the lookup of the shebang on the lang table, ;; and it will only happen if type-x is nil. ;; ‘type-f’ is the final type from all that, thus: ;; ;; - If a type you entered is found, the result is used. ;; - If not but you entered a type, this is used and assumed to be a ;; special block. Cases: ‘:type’ quote or ‘:type’ example. ;; - Otherwise we look at the .ext lookup. We use that if non-nil. ;; - Otherwise we try the shebang. If that is also not found but the ;; extension is “sh”, we use “shell”. ;; - If all that fails, there’s no type, and the contents will be ;; inserted without a src or special block. (type (org-reflect--ensure-string (plist-get params :type))) (type-t (when type (or (->> (assoc (read type) langs) cdr org-reflect--ensure-string) (->> (rassoc (read type) langs) cdr org-reflect--ensure-string)))) (type-x (unless type-t (when ext (->> (assoc (read ext) langs) cdr org-reflect--ensure-string)))) (type-s (unless type-x (if-let ((b (org-reflect--filetype-from-shebang file))) (->> (rassoc (read b) langs) cdr org-reflect--ensure-string) (when (string= "sh" ext) "shell")))) (type-f (or type-t (when type (if org-reflect-upcase-keywords-p (upcase type) type)) type-x type-s)) ;; Blocks (blk-p (and type-f (not (string= "no" (org-reflect--ensure-string (plist-get params :blk)))) t)) (blk-suf (when (or type-t (and (not type) (or type-x type-s))) src)) (blk-beg (when blk-p (concat beg "_" blk-suf (when blk-suf " ") type-f (when args " ") args "\n"))) (blk-end (when blk-p (concat end "_" (or blk-suf type-f)))) ;; Pre-reflection filters (el-cmd (plist-get params :el)) (el-fun (lambda () (when el-cmd (eval `(let ((s (buffer-string))) ;; ‘s’ should appear in el-cmd. (erase-buffer) (insert ,el-cmd)))))) (sh-cmd (--> (plist-get params :sh) (when it (format "%S" it)) ;; Should we bother stripping the list’s ;; parentheses? ;; ;; After all, what makes Elisp go “It’s a list!” ;; makes Bash go “It’s a subshell!” So it packs all ;; the commands inside it, ready for execution. ;; ;; Elegant coincidence. ;; ;; Although I expect the tiny hit in performance ;; from this extra subshelling to be unnoticeable, ;; I’ll remove them anyway. (org-reflect--unbracket-string "(" ")" it))) (sh-fun (lambda () (when sh-cmd (if (fboundp 'shell-command-on-region) (shell-command-on-region (point-min) (point-max) sh-cmd 'this-temp-buffer 'replace "*Org Reflect: Shell Filter Errors*") (error "Function %s is not available" 'shell-command-on-region))))) (res)) (with-temp-buffer (insert-file-contents file) (let* ((lmin (or (car (plist-get params :lr)) (plist-get params :lmin) 1)) (lmax (or (cadr (plist-get params :lr)) (plist-get params :lmax) (line-number-at-pos (point-max))))) (setq res (buffer-substring (point-at-bol lmin) (point-at-eol lmax))) (erase-buffer) ;; Make sure that at least one trailing newline is present, but ;; don’t mess with the string’s end otherwise. (insert (org-babel-chomp res "\n") "\n") ;; Pre-reflection filters ;; 1. If more than one type of the same filter is entered, all but ;; the first entry will be ignored. ;; 2. application order when more than one type is present is ;; determined by the order in which they appear in the header. (let ((filters (-uniq (-intersection params '(:el :sh))))) (dolist (filter filters) (pcase filter (:el (funcall el-fun)) (:sh (funcall sh-fun)) (_ (error "Error: this filter option shouldn’t happen"))))) ;; We don’t want to create spurious subtrees inside our block and ;; make it unparseable, right? (org-escape-code-in-region (point-min) (point-max)) ;; The results (setq res (concat blk-beg (org-reflect--normalize-trails (buffer-string) ;; Whether to add a trailing \n. When ;; reflection isn’t wrapped in an org src ;; or special org block, no \n is needed, ;; because the dynamic block will add ;; one. This will leave the text “tight” ;; into the reflection block. (not blk-p)) blk-end)))) (insert res)))) ;;;###autoload (defun org-reflect-update-all-dblocks () "Update all dblocks in the current file, including org-reflect’s. It makes sure that org-reflect has been loaded." (interactive) (require 'org-reflect) (org-update-all-dblocks)) ;;;###autoload (defun org-reflect-update-org-structure-template-alist () "Update ‘org-structure-template-alist’ for quick insertion. You’ll be able to add new ‘org-reflect’ blocks with ‘<keyTAB’, where ‘key’ is the value chosen in ‘org-reflect-org-structure-key’. See also Org Manual’s Info anchor ‘(org)Easy Templates’ \(Miscellaneous > Easy Templates)." (interactive) (let* ((key org-reflect-org-structure-key) (value (--> org-reflect-org-structure-value (if org-reflect-upcase-keywords-p (let ((upcased-alist (--map `(,it . ,(upcase it)) org-reflect-keywords))) ;; Note that the two "its" below do not refer ;; to the same thing as the two "its" above. (s-replace-all upcased-alist it)) it)))) (unless (and key value (add-to-list 'org-structure-template-alist `(,key ,value "")) (message "You can now add org-reflect blocks with: %s %s %s." "<" key "TAB")) (error "Couldn't update ‘org-structure-template-alist’")))) ;;;###autoload (defun org-reflect-show-special-expansions () "Show in a new buffer all the special expansions options for headers." (interactive) (let* ((orsx "*Org Reflect Special Expansions*") (sep (s-repeat (- (length orsx) 2) "=")) (buf (get-buffer-create orsx))) (switch-to-buffer buf) (erase-buffer) (insert ?\n) (emacs-lisp-mode) (->> org-reflect-args-expansions-special (-filter #'listp) (-flatten-n 1) (--map-when (= (length it) 2) (s-join " " (-map #'symbol-name it))) (--map (format "%s" it)) (s-join "\n") insert) (let ((buf-str (buffer-string))) (erase-buffer) (insert (org-reflect--unbracket-string "*" "*" orsx) ?\n sep ?\n "(see also those in ‘org-reflect-args-expansions-default’)" "\n\n\f\n" "Examples:" ?\n ":args \"$C0 $R1\" ==> :colnames no :rownames yes" ?\n ":args \"$t ~/o.el\" ==> :tangle ~/o.el" ?\n ":args $rvv ==> :results value verbatim" ?\n ":args $r--c ==> :results code" ?\n ":args $r-l-a ==> :results list append" ?\n ":args $r---s ==> :results silent" ?\n ":args $xb ==> :export both" ?\n "\f\n\n" "Special Expansions:" (s-replace "\n:" "\n\n:" buf-str) "\nType those (no abbreviations):\n" ":dir :sep :var :post") (goto-char (point-min)))))
;;;; Wrapping up (provide 'org-reflect) ;; Local Variables: ;; coding: utf-8 ;; indent-tabs-mode: nil ;; sentence-end-double-space: nil ;; outline-regexp: ";;;;* " ;; End: ;;; org-reflect.el ends here