Linux Kernel contains only C code?
Hi all, ... 1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?! 2. I believe some assembly language and shell scripting also used.. how can I check them? Can you point me which part of the source code contains the assembly and shell scripting Best regards Sekar
Hello Sekar, On Fri, Jan 26, 2018 at 3:58 PM, inventsekar <inventsekar@gmail.com> wrote:
Hi all, ...
1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
Yes, there are some scripts made in Perl, Python and other languages. Most of them are under tools/ or scripts/ or documentation/
2. I believe some assembly language and shell scripting also used.. how can I check them? Can you point me which part of the source code contains the assembly and shell scripting
https://github.com/torvalds/linux/search?l=assembly https://github.com/torvalds/linux/search?l=Shell I think you can see in the left down menu bar also other languages. Still, some language doesn't come out, but you can change assembly with Perl. You can also just find the file extensions with the find command or using git ls-files. find . -type f -name "*.pl" find . -type f -name "*.py" or git ls-files "*.pl" git ls-files "*.py" thanks, Alice -- Thanks, Alice Ferrazzi Gentoo Kernel Project Leader Gentoo Foundation Board Member Mail: Alice Ferrazzi <alice.ferrazzi@gmail.com> PGP: 2E4E 0856 461C 0585 1336 F496 5621 A6B2 8638 781A
On Fri, 26 Jan 2018 12:28:38 +0530, inventsekar said:
1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
Some of the userspace utilities are written in other languages, but the kernel itself is written in C and assembler.
2. I believe some assembly language and shell scripting also used.. how can I check them? Can you point me which part of the source code contains the assembly and shell scripting
C files have an extension .c, with possible short pieces of inline assembler with the gcc extension asm(). There's also C include files, which have names that end in .h Assembler files have an extension .s There's some shell scripting used for utilities and as part of the build process. In general, looking at the first few lines of code will tell you what language a file is, if the name doesn't end in .c or .s.
Hi Sekar,
Hi all, ... 1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
I have never seen c++, perl or python code in kernel source tree. Imagine that, if kernel relies on perl, python or other 3rd-party code, will it cause nested or mutual dependency issues? 3rd-party code bugs may also cause problems that make the kernel unstable.
2. I believe some assembly language and shell scripting also used.. how can I check them? Can you point me which part of the source code contains the assembly and shell scripting
Sorry, I do not know what do you mean by "check them". Assembly code are mostly put within the arch directory. They are used to interact with your various arch CPUs. In addition to arch directory, there are also some embedded assembly code that could be found in C source code. Shell scripts are always within tools and script directory. By the way pure assembly codes files ends with ".S" extension, and shell scriptes with ".sh" extension. You could find them by find ${kernel_source_dir} -name "*.S" and find ${kernel_source_dir} -name "*.sh" Regards Larry
On 01/26/2018 02:20 AM, Larry Chen wrote:
I have never seen c++, perl or python code in kernel source tree. Imagine that, if kernel relies on perl, python or other 3rd-party code, will it cause nested or mutual dependency issues? 3rd-party code bugs may also cause problems that make the kernel unstable.
Its not even really C, at least not as a normal application developer thinks of it. This is systems programming with a lot of kernel specific libraries. -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On Fri, 26 Jan 2018 05:53:20 -0500, Ruben Safir said:
On 01/26/2018 02:20 AM, Larry Chen wrote:
I have never seen c++, perl or python code in kernel source tree. Imagine that, if kernel relies on perl, python or other 3rd-party code, will it cause nested or mutual dependency issues? 3rd-party code bugs may also cause problems that make the kernel unstable.
Its not even really C, at least not as a normal application developer thinks of it. This is systems programming with a lot of kernel specific libraries.
It's C. The fact that things like stdio aren't available inside the kernel doesn't mean it's not C. (I've worked on large userspace code bases where you could go for several hundred thousand lines without seeing anything from /usr/include, and nobody would argue that code wasn't C.)
On Fri, Jan 26, 2018 at 2:12 PM, <valdis.kletnieks@vt.edu> wrote:
On Fri, 26 Jan 2018 05:53:20 -0500, Ruben Safir said:
Its not even really C, at least not as a normal application developer thinks of it. This is systems programming with a lot of kernel specific libraries.
It's C. The fact that things like stdio aren't available inside the kernel doesn't mean it's not C.
I would say that is C with GNU extensions [1]... Not plain/pure (ANSI/ISO) C. That's why it's not so easy to build the Linux Kernel with any other C compiler. There is a ongoing effort to build it with Clang [2]. I'm not sure how is the current status regarding the Intel C Compiler.
(I've worked on large userspace code bases where you could go for several hundred thousand lines without seeing anything from /usr/include, and nobody would argue that code wasn't C.)
Best regards, [1] https://www.ibm.com/developerworks/linux/library/l-gcc-hacks/ [2] https://lwn.net/Articles/734071/ -- Augusto Mecking Caringi
On Mon, Jan 29, 2018 at 10:41 AM, Augusto Mecking Caringi <augustocaringi@gmail.com> wrote:
On Fri, Jan 26, 2018 at 2:12 PM, <valdis.kletnieks@vt.edu> wrote:
On Fri, 26 Jan 2018 05:53:20 -0500, Ruben Safir said:
Its not even really C, at least not as a normal application developer thinks of it. This is systems programming with a lot of kernel specific libraries.
It's C. The fact that things like stdio aren't available inside the kernel doesn't mean it's not C.
I would say that is C with GNU extensions [1]... Not plain/pure (ANSI/ISO) C.
That's why it's not so easy to build the Linux Kernel with any other C compiler.
There is a ongoing effort to build it with Clang [2].
I'm not sure how is the current status regarding the Intel C Compiler.
Yeah, the kernel has too much undefined behavior and implementation defined behavior to build with other compilers at the moment. It would be nice if they moved away from UB and implementation defined behvior, but sometimes the political problems are heavier than the technical solutions. Jeff
On Mon, Jan 29, 2018 at 12:16:19PM -0500, Jeffrey Walton wrote:
On Mon, Jan 29, 2018 at 10:41 AM, Augusto Mecking Caringi <augustocaringi@gmail.com> wrote:
On Fri, Jan 26, 2018 at 2:12 PM, <valdis.kletnieks@vt.edu> wrote:
On Fri, 26 Jan 2018 05:53:20 -0500, Ruben Safir said:
Its not even really C, at least not as a normal application developer thinks of it. This is systems programming with a lot of kernel specific libraries.
It's C. The fact that things like stdio aren't available inside the kernel doesn't mean it's not C.
I would say that is C with GNU extensions [1]... Not plain/pure (ANSI/ISO) C.
That's why it's not so easy to build the Linux Kernel with any other C compiler.
There is a ongoing effort to build it with Clang [2].
I'm not sure how is the current status regarding the Intel C Compiler.
Yeah, the kernel has too much undefined behavior and implementation defined behavior to build with other compilers at the moment.
What other compilers do you suggest we use? Clang works great with the kernel, a few million phones are shipping with kernels built with it. icc has worked with the kernel for over a decade now, for anyone still stuck using that beast. What else is there?
It would be nice if they moved away from UB and implementation defined behvior, but sometimes the political problems are heavier than the technical solutions.
What do you mean by this? What "UB" does the kernel rely on that prevents compilers from properly building it? We do have some pretty strict requirements in the kernel for a compiler, but anything "odd" is usually just a bug and we are always willing to take patches to fix them. thanks, greg k-h
On Mon, Jan 29, 2018 at 12:33 PM, Greg KH <greg@kroah.com> wrote:
On Mon, Jan 29, 2018 at 12:16:19PM -0500, Jeffrey Walton wrote:
... It would be nice if they moved away from UB and implementation defined behvior, but sometimes the political problems are heavier than the technical solutions.
What do you mean by this? What "UB" does the kernel rely on that prevents compilers from properly building it? We do have some pretty strict requirements in the kernel for a compiler, but anything "odd" is usually just a bug and we are always willing to take patches to fix them.
https://groups.google.com/forum/#!topic/linux.kernel/-VzG6FWZiy8 and https://www.spinics.net/lists/linux-crypto/msg19466.html . I found the first one from 2013 amusing: JW >> According to Section 5.8, "Shift Operators" of JW >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2798.pdf: JW >> "The operands shall be of integral or enumeration type ... CL > The kernel doesn't try to be fully standard conformant. The second one from 2016 was disappointing. It suggested a standards compliant rotate that was constant time for the linux-crypto folks. That was rejected by some: PA >> So you are actually saying outright that we should sacrifice *actual* PA >> portability in favor of *theoretical* portability? What kind of PA >> twilight zone did we just step into?! Its kind of like I said... the political problems are harder than the technical solutions. Jeff
On Mon, Jan 29, 2018 at 01:00:10PM -0500, Jeffrey Walton wrote:
On Mon, Jan 29, 2018 at 12:33 PM, Greg KH <greg@kroah.com> wrote:
On Mon, Jan 29, 2018 at 12:16:19PM -0500, Jeffrey Walton wrote:
... It would be nice if they moved away from UB and implementation defined behvior, but sometimes the political problems are heavier than the technical solutions.
What do you mean by this? What "UB" does the kernel rely on that prevents compilers from properly building it? We do have some pretty strict requirements in the kernel for a compiler, but anything "odd" is usually just a bug and we are always willing to take patches to fix them.
https://groups.google.com/forum/#!topic/linux.kernel/-VzG6FWZiy8 and https://www.spinics.net/lists/linux-crypto/msg19466.html .
I found the first one from 2013 amusing:
JW >> According to Section 5.8, "Shift Operators" of JW >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2798.pdf: JW >> "The operands shall be of integral or enumeration type ... CL > The kernel doesn't try to be fully standard conformant.
The second one from 2016 was disappointing. It suggested a standards compliant rotate that was constant time for the linux-crypto folks. That was rejected by some:
PA >> So you are actually saying outright that we should sacrifice *actual* PA >> portability in favor of *theoretical* portability? What kind of PA >> twilight zone did we just step into?!
Well, let's not get into the crazy crypto code in the kernel, it's not a good example of much these days :)
Its kind of like I said... the political problems are harder than the technical solutions.
Those should now all be gone, thanks to some persistent developers. clang builds the kernel just fine as I mentioned, and if you look close, people are working to get something other than 'ld' to link it as well :) thanks, greg k-h
On Fri, Jan 26, 2018 at 1:58 AM, inventsekar <inventsekar@gmail.com> wrote: Hi all, ...
1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
Well, let's find out ? If you open up a shell/terminal and change into the top level directory of your Linux kernel source and run the command below: find . -type f -and -printf "%f\n" | grep -io '\.[^.]*$' | sort | uniq -c | sort -rn ( Breaking this down, find all files+get the filename+pull out the file extension+sort+only keep unique ext+sort with a stats count) On my system with the kernel source for Linux-4.15 it shows me what's below... keep in mind this is the number of 'unique' file extension types in the kernel source tree. It's long but please keep scrolling all the way there is useful stuff at the bottom. 25965 .c 20363 .h 4074 .txt 1470 .S 1437 .dts 1098 .dtsi 827 .rst 211 .gitignore 207 .json 200 .sh 75 .py 59 .cocci 57 .svg 49 .boot 42 .tc 39 .pl 36 .config 31 .debug 19 .lds 15 .conf 14 .ppm 14 .fuc 12 .fuc3 10 .exceptions 9 .y 9 .h_shipped 9 .c_shipped 9 .awk 8 .scr 8 .l 8 .html 8 .dot 8 .8 8 .1 7 .sa 7 .in 6 .cpp 6 .asn1 5 .xsl 5 .uc 5 .po 5 .inc 5 .fuc5 5 .cpu 4 .tbl 4 .S_shipped 4 .map 4 .ld 4 .include 4 .fail 4 .doc 3 .smp_defconfig 3 .pm 3 .mk 3 .gdbinit 3 .csv 3 .am 2 .up_defconfig 2 .um 2 .ubsan 2 .seq 2 .rules 2 .reg 2 .powerpc 2 .postlink 2 .platforms 2 .platform 2 .pbm 2 .nommu 2 .megaraid 2 .kasan 2 .inl 2 .inf 2 .ids 2 .fax 2 .FAQ 2 .build 2 .arm 1 .xs 1 .x86 1 .x25 1 .wimax 1 .WARNING 1 .vringh 1 .vim 1 .vdec2_defconfig 1 .vdec2 1 .uni 1 .txtd 1 .tex 1 .syncppp 1 .sym53c8xx_2 1 .sym53c8xx 1 .SRC 1 .soc 1 .smp 1 .select-break 1 .sed 1 .script 1 .sb1000 1 .rest 1 .recursion-issue-02 1 .recursion-issue-01 1 .readme 1 .README 1 .qlge 1 .qlcnic 1 .qla4xxx 1 .qla3xxx 1 .qla2xxx 1 .preempt 1 .PL 1 .perf 1 .pass 1 .ore 1 .opsp_defconfig 1 .normal 1 .nommu_defconfig 1 .netlink 1 .net 1 .ncr53c8xx 1 .modsign 1 .modpost 1 .modinst 1 .modes 1 .modbuiltin 1 .mm 1 .mISDN 1 .mips 1 .megaraid_sas 1 .md 1 .mak 1 .mailmap 1 .machine 1 .lpfc 1 .loopback 1 .locks 1 .Locking 1 .libfdt 1 .lib 1 .LIB 1 .kgdb 1 .kcov 1 .ipw2200 1 .ipw2100 1 .ips 1 .iosched 1 .inc_shipped 1 .ignore 1 .i2400m 1 .hz 1 .hysdn 1 .hp300 1 .host 1 .HiSax 1 .hfc-pci 1 .hex 1 .headersinst 1 .glade 1 .gitattributes 1 .gigaset 1 .gif 1 .generic 1 .gdbinit_400MHz_32MB 1 .gdbinit_300MHz_32MB 1 .gdbinit_200MHz_16MB 1 .gcc-plugins 1 .gate 1 .fuc4 1 .fuc0s 1 .freezer 1 .FPE 1 .FlashPoint 1 .feature 1 .extrawarn 1 .example 1 .dtc 1 .dtbinst 1 .DOC 1 .diversion 1 .dino 1 .devices 1 .default 1 .def 1 .DAC960 1 .cycladesZ 1 .css 1 .cputype 1 .copyright 1 .concap 1 .common 1 .cocciconfig 1 .clean 1 .checkpatch 1 .char 1 .ChangeLog 1 .cfg 1 .cert 1 .cc 1 .CAPI 1 .cache 1 .bus 1 .buddha 1 .binfmt 1 .bc 1 .avmb1 1 .audio 1 .asm-generic 1 .asm 1 .arcmsr 1 .arch 1 .aic7xxx 1 .aic79xx 1 .agh 1 .ac 1 .1996-2002 1 .1995-2002 1 .1994-2004 1 .1992-1997 1 . Somethings are obvious like we have : 25965 .c files 20363 .h files 4074 .txt files and 1470 .S files. Somethings are not so obvious like what could possibly be a *.y file or *.tc file ? If you type in find -name "*.y" in my case i see: aruna@debian:~/linux-4.15$ find -name "*.y" ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y ./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y ./tools/perf/util/expr.y ./tools/perf/util/pmu.y ./tools/perf/util/parse-events.y ./tools/bpf/bpf_exp.y ./scripts/dtc/dtc-parser.y ./scripts/genksyms/parse.y ./scripts/kconfig/zconf.y Now if I pass that to the 'file' command ... aruna@debian:~/linux-4.15$ file `find -name "*.y"` // yes you need those back ticks :) ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y: C source, ASCII text ./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y: C source, ASCII text ./tools/perf/util/expr.y: C source, ASCII text ./tools/perf/util/pmu.y: C source, ASCII text ./tools/perf/util/parse-events.y: C source, ASCII text ./tools/bpf/bpf_exp.y: C source, ASCII text ./scripts/dtc/dtc-parser.y: C source, ASCII text ./scripts/genksyms/parse.y: C source, ASCII text ./scripts/kconfig/zconf.y: C source, ASCII text So 'file' tells us these are C program files ? Lets verify ? If you 'cat' any of these files you will see it is actual C code. Why does it have a file extension of .y ? Valdis or Gregkh may be able to shed some light on that I have no idea :) Now we do the exact same thing for *.tc files. aruna@debian:~/linux-4.15$ find -name "*.tc" <snip> ./tools/testing/selftests/ftrace/test.d/00basic/basic3.tc ./tools/testing/selftests/ftrace/test.d/00basic/basic4.tc <snip> and pass this to 'file' and we see.. they seem to be shell scripts ? aruna@debian:~/linux-4.15$ file `find -name "*.tc"` <snip> ./tools/testing/selftests/ftrace/test.d/00basic/basic3.tc: POSIX shell script, ASCII text executable ./tools/testing/selftests/ftrace/test.d/00basic/basic4.tc: POSIX shell script, ASCII text executable <snip> So let's verify again ? By running -->aruna@debian:~/linux-4.15$ cat ./tools/testing/selftests/ftrace/test.d/00basic/basic4.tc and we see they are shell scripts. #!/bin/sh # description: Basic event tracing check test -f available_events -a -f set_event -a -d events # check scheduler events are available grep -q sched available_events && exit_pass || exit_fail There are many 'interesting' things hidden away down the kernel source rabbit hole. So explore with a child's curiosity, have lot's of patience and wash +rinse+repeat as necessary. Good luck ! Aruna ( to those who are long time kernel hackers and veterans my humble apologies but I felt this was needed.. )
2. I believe some assembly language and shell scripting also used.. how can I check them? Can you point me which part of the source code contains the assembly and shell scripting
Best regards Sekar
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Thu, Feb 1, 2018 at 5:37 PM, Aruna Hewapathirane <aruna.hewapathirane@gmail.com> wrote:
On Fri, Jan 26, 2018 at 1:58 AM, inventsekar <inventsekar@gmail.com> wrote: Hi all, ...
1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
Well, let's find out ? If you open up a shell/terminal and change into the top level directory of your Linux kernel source and run the command below:
find . -type f -and -printf "%f\n" | grep -io '\.[^.]*$' | sort | uniq -c | sort -rn ( Breaking this down, find all files+get the filename+pull out the file extension+sort+only keep unique ext+sort with a stats count)
For that I recommend a tool called sloccount [1]... BTW, running it now against Linux Kernel source I got: Totals grouped by language (dominant language first): ansic: 16675070 (97.83%) asm: 294179 (1.73%) perl: 26346 (0.15%) sh: 18781 (0.11%) python: 15642 (0.09%) cpp: 6512 (0.04%) yacc: 4586 (0.03%) lex: 2479 (0.01%) awk: 1387 (0.01%) pascal: 231 (0.00%) sed: 5 (0.00%) Total Physical Source Lines of Code (SLOC) = 17,045,218 [1] https://www.dwheeler.com/sloccount/ -- Augusto Mecking Caringi
On Thu, Feb 01, 2018 at 05:49:13PM +0100, Augusto Mecking Caringi wrote:
On Thu, Feb 1, 2018 at 5:37 PM, Aruna Hewapathirane <aruna.hewapathirane@gmail.com> wrote:
On Fri, Jan 26, 2018 at 1:58 AM, inventsekar <inventsekar@gmail.com> wrote: Hi all, ...
1. May i know, other than C language, is there any other programming language is/are used inside Linux Kernel?!?! is there any c++, Perl, python programs are used for peculiar tasks inside Linux Kernel?!?!
Well, let's find out ? If you open up a shell/terminal and change into the top level directory of your Linux kernel source and run the command below:
find . -type f -and -printf "%f\n" | grep -io '\.[^.]*$' | sort | uniq -c | sort -rn ( Breaking this down, find all files+get the filename+pull out the file extension+sort+only keep unique ext+sort with a stats count)
For that I recommend a tool called sloccount [1]...
BTW, running it now against Linux Kernel source I got:
Totals grouped by language (dominant language first): ansic: 16675070 (97.83%) asm: 294179 (1.73%) perl: 26346 (0.15%) sh: 18781 (0.11%) python: 15642 (0.09%) cpp: 6512 (0.04%) yacc: 4586 (0.03%) lex: 2479 (0.01%) awk: 1387 (0.01%) pascal: 231 (0.00%) sed: 5 (0.00%) Total Physical Source Lines of Code (SLOC) = 17,045,218
'tokei' is _much_ faster when dealing with large code bases than sloccount. And gives a bit different view of the tree as well, I'd recommend it over sloccount these days. thanks, greg k-h
On Thu, 01 Feb 2018 11:37:26 -0500, Aruna Hewapathirane said:
Somethings are not so obvious like what could possibly be a *.y file or *.tc file ? If you type in find -name "*.y" in my case i see:
aruna@debian:~/linux-4.15$ find -name "*.y"
Now if I pass that to the 'file' command ...
aruna@debian:~/linux-4.15$ file `find -name "*.y"` // yes you need those back ticks :)
./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y: C source, ASCII text
So 'file' tells us these are C program files ? Lets verify ? If you 'cat' any of these files you will see it is actual C code. Why does it have a file extension of .y ?
Actually, if you look more closely at it, it's *not* actually C code. The 'file' command makes its best guess, triggered off things like '#include', etc. The 'tl;dr' answer is "The *.y are input files for bison, and the *.l are input files for flex". The more detailed explanation, with 50 years of computer history.... [/usr/src/linux-next] head -20 tools/perf/util/expr.y /* Simple expression parser */ %{ #include "util.h" #include "util/debug.h" #define IN_EXPR_Y 1 #include "expr.h" #include "smt.h" #include <string.h> #define MAXIDLEN 256 %} %pure-parser %parse-param { double *final_val } %parse-param { struct parse_ctx *ctx } %parse-param { const char **pp } %lex-param { const char **pp } %union { double num; That's got a bunch of % in unusual places for C, doesn't it? :) Let's hit the rewind button back 5 decades or so, when tools for building programs were just becoming created. And everybody who wanted to write a compiler for a language, or parsing data that wasn't strict 'ID is in columns 5-12' formatting, or a whole bunch of other stuff, had to write a parser to do the parsing. For those who have never done it, writing lexical scanners and parsers by hand is a thankless job. I know from experience that the parse table for an LALR parser for Pascal ends up being essentially a spreadsheet with some 300 rows and 400 columns that you get to fill in by hand one at a time - and getting one entry wrong means you have a buggy compiler (I took Compiler Design in college the last year we had to do it by hand) The first few compiled languages (COBOL, FORTRAN, and a few others) also had to make do with hand-coded parsers. And then in 1958, Algol happened, and it spawned all sorts of languages - everything from C to PL/I to Pascal and probably 200+ others (pretty much every language that allows nested declarations and has begin/end tokens of some sort owes it to Algol). And the other thing about Algol was that it was a much "bigger" language than previous ones, so John Backus invented a meta-language called BNF to provide a formal specification of the syntax. (For those who are curious, a EBNF specification for Pascal syntax is here: http://www.fit.vutbr.cz/study/courses/APR/public/ebnf.html The interesting thing about BNF is that it has these things called "production rules" which define what legal programs look like - and the test for "legal" can be done with a parser using a software/mathematical construct called a "finite state machine" (and the 3 of you who understand the difference between a context-sensitive grammar and a context-free grammar can stop giggling right now.. ;) So somebody had the bright idea that if you had a formal BNF specification, you could write a program that would read the BNF, and spit out the C source for a parser skeleton based on a finite state machine. And thus were born two programs called 'lex' (a lexical scanner - something that reads the source, and outputs code that says "Hey, that's the word 'struct'" or "we just saw a 'for"). and another called 'yacc' (Yet Another Compiler Compiler) which did the higher level "this is a legal function, but *that* right there is a messed-up 'if' statement that has a syntax error" stuff. Oh, and generate output code, too. Of course, that was decades ago, and eventually somebody wrote a faster 'lex' - and thus was born /usr/bin/flex. And yacc needed work, so the improved version was, of course, called bison (think about it for a bit..)
I'm just curious. What problems in kernel involves parsing? On Feb 2, 2018 5:01 AM, <valdis.kletnieks@vt.edu> wrote:
On Thu, 01 Feb 2018 11:37:26 -0500, Aruna Hewapathirane said:
Somethings are not so obvious like what could possibly be a *.y file or *.tc file ? If you type in find -name "*.y" in my case i see:
aruna@debian:~/linux-4.15$ find -name "*.y"
Now if I pass that to the 'file' command ...
aruna@debian:~/linux-4.15$ file `find -name "*.y"` // yes you need those back ticks :)
./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y: C source, ASCII text
So 'file' tells us these are C program files ? Lets verify ? If you 'cat' any of these files you will see it is actual C code. Why does it have a file extension of .y ?
Actually, if you look more closely at it, it's *not* actually C code. The 'file' command makes its best guess, triggered off things like '#include', etc.
The 'tl;dr' answer is "The *.y are input files for bison, and the *.l are input files for flex".
The more detailed explanation, with 50 years of computer history....
[/usr/src/linux-next] head -20 tools/perf/util/expr.y /* Simple expression parser */ %{ #include "util.h" #include "util/debug.h" #define IN_EXPR_Y 1 #include "expr.h" #include "smt.h" #include <string.h>
#define MAXIDLEN 256 %}
%pure-parser %parse-param { double *final_val } %parse-param { struct parse_ctx *ctx } %parse-param { const char **pp } %lex-param { const char **pp }
%union { double num;
That's got a bunch of % in unusual places for C, doesn't it? :)
Let's hit the rewind button back 5 decades or so, when tools for building programs were just becoming created. And everybody who wanted to write a compiler for a language, or parsing data that wasn't strict 'ID is in columns 5-12' formatting, or a whole bunch of other stuff, had to write a parser to do the parsing.
For those who have never done it, writing lexical scanners and parsers by hand is a thankless job. I know from experience that the parse table for an LALR parser for Pascal ends up being essentially a spreadsheet with some 300 rows and 400 columns that you get to fill in by hand one at a time - and getting one entry wrong means you have a buggy compiler (I took Compiler Design in college the last year we had to do it by hand)
The first few compiled languages (COBOL, FORTRAN, and a few others) also had to make do with hand-coded parsers. And then in 1958, Algol happened, and it spawned all sorts of languages - everything from C to PL/I to Pascal and probably 200+ others (pretty much every language that allows nested declarations and has begin/end tokens of some sort owes it to Algol). And the other thing about Algol was that it was a much "bigger" language than previous ones, so John Backus invented a meta-language called BNF to provide a formal specification of the syntax.
(For those who are curious, a EBNF specification for Pascal syntax is here: http://www.fit.vutbr.cz/study/courses/APR/public/ebnf.html
The interesting thing about BNF is that it has these things called "production rules" which define what legal programs look like - and the test for "legal" can be done with a parser using a software/mathematical construct called a "finite state machine" (and the 3 of you who understand the difference between a context-sensitive grammar and a context-free grammar can stop giggling right now.. ;)
So somebody had the bright idea that if you had a formal BNF specification, you could write a program that would read the BNF, and spit out the C source for a parser skeleton based on a finite state machine. And thus were born two programs called 'lex' (a lexical scanner - something that reads the source, and outputs code that says "Hey, that's the word 'struct'" or "we just saw a 'for"). and another called 'yacc' (Yet Another Compiler Compiler) which did the higher level "this is a legal function, but *that* right there is a messed-up 'if' statement that has a syntax error" stuff. Oh, and generate output code, too.
Of course, that was decades ago, and eventually somebody wrote a faster 'lex' - and thus was born /usr/bin/flex. And yacc needed work, so the improved version was, of course, called bison (think about it for a bit..)
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Fri, Feb 2, 2018 at 11:11 AM, Daniel. <danielhilst@gmail.com> wrote:
I'm just curious. What problems in kernel involves parsing?
$ find -name *.[yl] ./drivers/scsi/aic7xxx/aicasm/aicasm_scan.l ./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_scan.l ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y ./tools/perf/util/pmu.l ./tools/perf/util/expr.y ./tools/perf/util/parse-events.y ./tools/perf/util/parse-events.l ./tools/perf/util/pmu.y ./tools/bpf/bpf_exp.y ./tools/bpf/bpf_exp.l ./scripts/genksyms/parse.y ./scripts/genksyms/lex.l ./scripts/dtc/dtc-lexer.l ./scripts/dtc/dtc-parser.y ./scripts/kconfig/zconf.l ./scripts/kconfig/zconf.y So, really in the kernel just a SCSI driver that needs to parse a "Host adapter sequencer assembler"... The others *[yl] files are in tools or scripts directories (userspace tools like perf or build support scripts). -- Augusto Mecking Caringi
Oh, thank you so much! I see dtc, the device tree compiler I guess. Parsing is a cool programming subject but I didn't expect to see parsers in the kernel level, cool, cool! Regards, 2018-02-02 8:27 GMT-02:00 Augusto Mecking Caringi <augustocaringi@gmail.com> :
On Fri, Feb 2, 2018 at 11:11 AM, Daniel. <danielhilst@gmail.com> wrote:
I'm just curious. What problems in kernel involves parsing?
$ find -name *.[yl] ./drivers/scsi/aic7xxx/aicasm/aicasm_scan.l ./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_scan.l ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y ./tools/perf/util/pmu.l ./tools/perf/util/expr.y ./tools/perf/util/parse-events.y ./tools/perf/util/parse-events.l ./tools/perf/util/pmu.y ./tools/bpf/bpf_exp.y ./tools/bpf/bpf_exp.l ./scripts/genksyms/parse.y ./scripts/genksyms/lex.l ./scripts/dtc/dtc-lexer.l ./scripts/dtc/dtc-parser.y ./scripts/kconfig/zconf.l ./scripts/kconfig/zconf.y
So, really in the kernel just a SCSI driver that needs to parse a "Host adapter sequencer assembler"...
The others *[yl] files are in tools or scripts directories (userspace tools like perf or build support scripts).
-- Augusto Mecking Caringi
-- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski
participants (10)
-
Alice Ferrazzi -
Aruna Hewapathirane -
Augusto Mecking Caringi -
Daniel. -
Greg KH -
inventsekar -
Jeffrey Walton -
Larry Chen -
Ruben Safir -
valdis.kletnieks＠vt.edu