| Commit message (Expand) | Author | Age | Files | Lines |
|---|
| * | http: Add /api/evaluations route.•••* src/cuirass/database.scm (db-get-evaluations): New exported procedure.
* src/cuirass/http.scm (url-handler): Add /api/evaluations route.
* tests/http.scm ("http"): Add /api/evaluations test route.
| Mathieu Othacehe | 2018-04-19 | 2 | -0/+26 |
| * | database: Adjust index to speed up /api/queue queries.•••* src/schema.sql: Extend 'Builds_Derivations_index' to account for
/api/queue queries.
| Ludovic Courtès | 2018-04-10 | 1 | -2/+2 |
| * | base: 'spawn-builds' really builds by chunks.•••Fixes a regression introduced in
074b9d02f1ca01007f39adbc019763027a51d9bd whereby we'd attempt to build
all of DRV at once.
* src/cuirass/base.scm (spawn-builds): Pass BATCH, not DRV, to
'build-derivations&' and to 'update-build-statuses!'.
| Ludovic Courtès | 2018-04-09 | 1 | -2/+2 |
| * | base: Pass the correct load path to the 'evaluate' command.•••The previous load path was potentially incorrect since commit
2fe7ff87e23b18d49bd33cffc4766b7eaa382054.
* src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables.
Assume #:load-path is colon-separated. Pass LOAD-PATH as the second
argument to 'evaluate'.
* doc/cuirass.texi (Database): Adjust documentation.
| Ludovic Courtès | 2018-04-07 | 1 | -5/+17 |
| * | base: Let sqlite handle deduplication of the list of pending derivations.•••Previously we would make a SQL query that would return many build jobs,
and then call 'delete-duplicates' on that. This was extremely wasteful
because the list of returned by the query was huge leading to a heap of
several tens of GiB on a big database, and 'delete-duplicates' would
lead to more GC and it would take ages.
Furthermore, since 'delete-duplicates' is written in C as of Guile
2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the
kernel thread running the 'restart-builds' fiber would never schedule
other fibers, which could lead to deadlocks--e.g., since fibers are
scheduled on a circular shuffled list of kernel threads, once every N
times, a web server fiber would be sent to that kernel thread and not be
serviced.
* src/cuirass/base.scm (shuffle-jobs): Remove.
(shuffle-derivations): New procedure.
(spawn-builds): Take a list of derivations instead of a list of jobs.
(restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates'
call. Remove done/remaining partitioning.
(build-packages): Adjust to pass 'spawn-builds' a list of derivations.
* bin/cuirass.in (main): Remove computation of PENDING. Remove second
parameter in call to 'restart-builds'.
| Ludovic Courtès | 2018-04-05 | 1 | -54/+28 |
| * | database: Add 'db-get-pending-derivations'.•••* src/cuirass/database.scm (db-get-pending-derivations): New procedure.
* tests/database.scm ("database")["db-get-pending-derivations"]: New test.
| Ludovic Courtès | 2018-04-05 | 1 | -0/+17 |
| * | utils: Add critical sections.•••* src/cuirass/utils.scm (make-critical-section)
(call-with-critical-section): New procedures.
(with-critical-section): New macro.
* src/cuirass/http.scm (with-database-access): Remove.
(handle-build-request, handle-builds-request, url-handler): Use
'with-critical-section' instead of 'with-database-access'.
(run-cuirass-server): Remove 'spawn-fiber' call. Use
'make-critical-section' instead.
| Ludovic Courtès | 2018-04-02 | 2 | -30/+44 |
| * | base: Do not resort to Coreutils' "chmod".•••* src/cuirass/base.scm (make-writable-copy)[chmod+w]: New procedure.
Replace 'system*' call with 'file-system-fold' call.
| Ludovic Courtès | 2018-04-01 | 1 | -1/+15 |
| * | base: Make a writable copy of the checkout only when #:no-compile? is false.•••This avoids copying things back and forth.
* src/cuirass/base.scm (fetch-repository): Add #:writable-copy?
parameter. Call 'make-writable-copy' when it's true.
(copy-repository-cache): Remove.
(make-writable-copy): New procedure.
(evaluate): Add 'source' parameter and pass it to the 'evaluate' program.
(process-specs): Define 'compile?'. Pass #:writable-copy? to
'fetch-repository'. Remove call to 'copy-repository-cache'. Remove
computation of the checkout directory name. Pass CHECKOUT to 'evaluate'.
* bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove
computation of the checkout directory name.
| Ludovic Courtès | 2018-04-01 | 1 | -28/+39 |
| * | database: 'db-get-builds' honors 'status+submission-time' ordering again.•••Fixes a regression introduced in
1bab5c4e56eb1849edc2cf0b23d433aeb2cac421 whereby the
'status+submission-time' order would no longer be honored.
As a result, /api/queue would return the queue ordered by build IDs,
making it largely useless.
* src/cuirass/database.scm (db-get-builds): Remove 'order' and rename
'order-column-name' to 'order'. Add case for 'status+submission-time'.
* tests/database.scm ("database")["db-get-builds"]: Move below
"db-update-build-status!" test. Add case for the
'status+submission-time' order.
| Ludovic Courtès | 2018-03-29 | 1 | -12/+12 |
| * | http: Process client connections really concurrently, again.•••This reinstates c47dfdf82b4be62501a7932eaec4c124566a1829 and fixes the
issues that led to the revert in
b71f0cdca5aeb82e5eb24f54b32e3f09fee22bad.
Before that, 'run-server' would force sequential processing of client
requests one after another.
* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'. Spawn a database fiber.
(with-database-access): New macro.
(handle-build-request): Expect 'db-channel' and use 'with-database-access'.
(handle-builds-request): Likewise.
(url-handler): Likewise.
| Ludovic Courtès | 2018-03-28 | 1 | -32/+79 |
| * | base: Add 'cancel-old-builds'.•••* src/cuirass/base.scm (cancel-old-builds): New procedure.
| Ludovic Courtès | 2018-03-28 | 1 | -0/+9 |
| * | database: Set a 'busy_timeout' to handle concurrent accesses.•••Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while
accessing the database: see
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>.
Suggested by Danny Milosavljevic <dannym@scratchpost.org>.
* src/cuirass/database.scm (wal-mode): Rename to...
(set-db-options): ... this. Add call to 'sqlite-exec' for
'busy_timeout'.
| Ludovic Courtès | 2018-03-25 | 1 | -5/+15 |
| * | 'with-store' and 'with-database' and written in terms of 'unwind-protect'.•••* src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'.
* src/cuirass/database.scm (with-database): Likewise.
| Ludovic Courtès | 2018-03-19 | 2 | -17/+14 |
| * | utils: Add 'unwind-protect'.•••* src/cuirass/utils.scm (unwind-protect): New macro.
| Ludovic Courtès | 2018-03-19 | 1 | -0/+20 |
| * | http: Correctly set #:timestamp for non-finished builds.•••Fixes a regression introduced in
f083282fd3bf813fda0b54ed33278d2d5325dfa1, whereby we'd return 0 as the
timestamp for everything in /api/queue.
* src/cuirass/http.scm (build->hydra-build): Make 'finished?' a
Boolean. Move 'bool->int' call in #:finished definition.
| Ludovic Courtès | 2018-03-18 | 1 | -5/+4 |
| * | base: Catch errors in the 'process-build-log' handler.•••Previously, when an exception was raised from
'handle-build-event' (e.g., a "database is locked" error), we'd throw,
thereby leaving PORT open and we'd never read from it again. Thus, the
corresponding 'guix-daemon' process would eventually get stuck in a
'write' call to that socket, and its build processes would stall.
* src/cuirass/base.scm (exception-reporter): New procedure.
(spawn-builds): Use it.
| Ludovic Courtès | 2018-03-18 | 1 | -1/+18 |
| * | base: Move database update from 'evaluate' process to the main process.•••Fixes <https://bugs.gnu.org/30618>.
Reported by Andreas Enge <andreas@enge.fr>.
* bin/evaluate.in (fill-job): Remove.
(main): Remove 'database' command-line argument. Remove DB and its
uses. Write an (evaluation EVAL JOBS) sexp.
* src/cuirass/base.scm (evaluate)[augment-job]: New procedure.
Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call
'db-add-evaluation' and 'db-add-derivation'.
| Ludovic Courtès | 2018-03-01 | 1 | -12/+29 |
| * | base: 'spawn-builds' correctly keeps track of remaining builds.•••Fixes <https://bugs.gnu.org/30645>.
Reported by Andreas Enge <andreas@enge.fr>.
* src/cuirass/base.scm (spawn-builds): Fix TOTAL vs. COUNT mismatches.
| Ludovic Courtès | 2018-03-01 | 1 | -3/+3 |
| * | http: Return build completion time as #:timestamp when completed.•••* src/cuirass/http.scm (build->hydra-build): Set #:timestamp to
#:stoptime when BUILD is finished.
| Ludovic Courtès | 2018-03-01 | 1 | -5/+14 |
| * | database: Indent 'db-get-builds'.•••* src/cuirass/database.scm (db-get-builds): Reindent.
| Ludovic Courtès | 2018-03-01 | 1 | -49/+49 |
| * | database: Adjust style of 'assqx-ref'.•••* src/cuirass/database.scm (db-get-builds)[assqx-ref]: Rewrite with
'match'.
| Ludovic Courtès | 2018-03-01 | 1 | -6/+7 |
| * | http: 'request-parameters' always returns a list.•••* src/cuirass/http.scm (request-parameters): Return the empty list when
QUERY is #f.
| Ludovic Courtès | 2018-03-01 | 1 | -11/+12 |
| * | http: Fix interpretation of non-numerical parameters.•••Fixes a regression introduced in
593cb7be108ed97bca5371aad2e53fa8ce4817ba.
* src/cuirass/http.scm (request-parameters): Fix fallback case in
'match' form. Previously it would return a procedure in this case, as
returned by (const param), leading to a failure down the road in
'sqlite-bind-arguments' as could be seen by running tests/http.scm.
| Ludovic Courtès | 2018-03-01 | 1 | -4/+5 |
| * | logging: "Defensive programming" for 'log-monitoring-stats'.•••I've seen 'scandir' report #f once, even though that's theoretically
impossible.
* src/cuirass/logging.scm (log-monitoring-stats): Return '() if
'scandir' returns #f.
| Ludovic Courtès | 2018-02-27 | 1 | -3/+6 |
| * | database: Fix grouping in db-get-builds.•••* src/cuirass/database.scm (db-get-builds): Fix grouping.
| Danny Milosavljevic | 2018-02-19 | 1 | -12/+5 |
| * | http: Interpret id and nr request-parameters as numbers.•••* src/cuirass/http.scm (request-parameters): Interpret id and nr parameters as
numbers.
| Danny Milosavljevic | 2018-02-19 | 1 | -1/+5 |
| * | database: db-get-builds: Remove debugging output.•••* src/cuirass/database.scm (db-get-builds): Remove debugging output.
(db-get-build): Remove debugging output.
| Danny Milosavljevic | 2018-02-19 | 1 | -9/+0 |
| * | http: Convert build-id URL part to number.•••* src/cuirass/http.scm (url-handler): Convert build-id URL part to number.
| Danny Milosavljevic | 2018-02-19 | 1 | -2/+2 |
| * | database: db-get-builds: Inline output selection.•••* src/cuirass/database.scm (db-get-builds): Inline output selection.
| Danny Milosavljevic | 2018-02-19 | 1 | -4/+93 |
| * | database: Simplify 'db-get-builds'.•••* src/cuirass/database.scm (db-get-builds): Modify.
(db-get-build): Modify.
| Danny Milosavljevic | 2018-02-19 | 1 | -109/+54 |
| * | cuirass: Clear the build queue when starting.•••* src/cuirass/base.scm (clear-build-queue): New procedure.
* bin/cuirass.in (main): Call it.
| Ludovic Courtès | 2018-02-14 | 1 | -0/+7 |
| * | sql: Add indices to speed up common queries.•••* src/schema.sql: Add indices.
| Ludovic Courtès | 2018-02-14 | 1 | -0/+6 |
| * | Revert "http: Process client connections really concurrently."•••This reverts commit c47dfdf82b4be62501a7932eaec4c124566a1829.
Processing connections concurrently would require having separate
database handles. See
<https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00206.html>.
| Ludovic Courtès | 2018-02-14 | 1 | -25/+8 |
| * | database: Make 'db-add-derivation' idempotent.•••* src/cuirass/database.scm (db-add-derivation): Catch 'sqlite-error and
handle SQLITE_CONSTRAINT_PRIMARYKEY.
(SQLITE_CONSTRAINT_UNIQUE): New variable.
* tests/database.scm ("database")["db-add-derivation"]: Add extra call to
'db-add-derivation'.
| Ludovic Courtès | 2018-02-14 | 1 | -7/+19 |
| * | http: Process client connections really concurrently.•••Before that, 'run-server' would force sequential processing of client
requests one after another.
* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'.
| Ludovic Courtès | 2018-02-10 | 1 | -5/+26 |
| * | utils: 'non-blocking' forwards exceptions to the calling fiber.•••* src/cuirass/utils.scm (%non-blocking): Forward exceptions to the
calling fiber.
| Ludovic Courtès | 2018-02-08 | 1 | -4/+13 |
| * | database: Use argument binding in 'db-get-builds' queries.•••That makes it safe from SQL injection.
* src/cuirass/database.scm (db-get-builds): Rewrite to use question
marks in SQL queries and binding through '%sqlite-exec'.
* tests/database.scm ("database")["db-get-builds"]: Exercise 'WHERE'
clauses.
| Ludovic Courtès | 2018-02-08 | 1 | -44/+65 |
| * | database: Handle binding directly in 'sqlite-exec'.•••The new macro automatically takes care of inserting question marks in
the SQL queries, which in turn guarantees that there are always as many
question marks and arguments.
* src/cuirass/database.scm (sqlite-exec): Rename to...
(%sqlite-exec): ... this.
(sqlite-exec/bind, sqlite-exec): New macros.
(assq-refs): Remove.
(db-add-specification): Use the new 'sqlite-exec' form.
(db-get-specifications): Correctly deal with REV or TAG being #f.
(db-add-derivation, db-get-derivation, db-add-evaluation)
(db-add-build, db-update-build-status!, db-get-outputs)
(db-get-build, db-get-stamp, db-add-stamp): Adjust to the new
'sqlite-exec' form.
| Ludovic Courtès | 2018-02-08 | 1 | -57/+101 |
| * | base: Account for derivations built behind our back.•••Previously any derivation not directly built by Cuirass would be
considered as failed because 'handle-build-event' wouldn't see any build
event. Here we just make sure the build status recorded in the database
corresponds to reality.
* src/cuirass/base.scm (update-build-statuses!): New procedure.
(spawn-builds): Call it after 'build-derivations&'.
| Ludovic Courtès | 2018-02-08 | 1 | -8/+31 |
| * | base: Fix computation of build success/failure.•••* src/cuirass/base.scm (build-packages)[register]: Make 'db-add-build' a
tail call.
Fix computation of 'outs'.
| Ludovic Courtès | 2018-02-08 | 1 | -3/+5 |
| * | database: 'db-update-build-status!' keeps stoptime unchanged when nothing new.•••* src/cuirass/database.scm (db-update-build-status!): And "AND status != ?"
in SQL queries.
* tests/database.scm ("database")["db-update-build-status!"]: Add call
to 'db-update-build-status!'.
| Ludovic Courtès | 2018-02-08 | 1 | -4/+9 |
| * | base: Explicitly enable #:print-build-trace.•••This is purely "defensive programming" since the default value for
'print-build-trace' is currently #t.
* src/cuirass/base.scm (with-store): Pass #:print-build-trace to
'set-build-options'.
| Ludovic Courtès | 2018-02-08 | 1 | -2/+4 |
| * | database: Cache prepared statements.•••Suggested by Danny Milosavljevic.
* src/cuirass/database.scm (sqlite-exec): Pass #:cache? to
'sqlite-prepare'.
| Ludovic Courtès | 2018-02-08 | 1 | -1/+1 |
| * | database: Use 'sqlite-bind' to avoid SQL injection.•••* src/cuirass/database.scm (%sqlite-exec): Remove.
(sqlite-exec): Turn back into a procedure. Use 'sqlite-bind'. Add
'normalize' procedure and use it.
(db-add-specification, db-add-derivation, db-get-derivation)
(db-add-evaluation, db-add-build, db-update-build-status!)
(db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL
queries.
* src/cuirass/base.scm (build-packages)[register]: Make #:log
non-false.
* tests/database.scm (make-dummy-job): Add #:job-name, #:system,
#:nix-name, and #:eval-id. This is necessary because 'sqlite-bind'
would now translate #f to a real NULL (before it would translate to the
string "#f"...), and would thus report violations of the non-NULL
constraint.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
| Danny Milosavljevic | 2018-02-08 | 2 | -41/+46 |
| * | base: Remove useless 'log-message' calls.•••* src/cuirass/base.scm (build-packages): Remove useless 'log-message'
calls.
| Ludovic Courtès | 2018-02-05 | 1 | -2/+0 |
| * | http: /jobsets returns a list of jobsets.•••Reported by Danny Milosavljevic.
* src/cuirass/http.scm (url-handler) <"jobsets"> Return the result of
'db-get-specifications' as-is, not just the car.
| Ludovic Courtès | 2018-02-05 | 1 | -1/+1 |
| * | base: Log the number of builds performed.•••* src/cuirass/base.scm (spawn-builds): Log the number of builds
performed, not the number of remaining builds.
| Ludovic Courtès | 2018-02-05 | 1 | -1/+1 |
| * | base: Work around Fibers I/O scheduling bug.•••Works around <https://github.com/wingo/fibers/issues/19>.
The effect in practice would be that we'd usually not reach the
'close-pipe' call in 'evaluate', leaving zombie processes behind us,
never executing the continuation, and additionally spinning fast on a
sequence of epoll_wait/epoll_ctl calls.
* src/cuirass/base.scm <top level>: Monkey-patch (fibers internal).
| Ludovic Courtès | 2018-01-30 | 1 | -0/+10 |
| * | base: Make build log processing non-blocking.•••We used to have 'build-derivations' write to the custom binary port
returned by 'build-event-output-port'. However, custom binary ports
constitute continuation barriers, thereby preventing fibers from being
suspended.
To make build log processing non-blocking, we therefore invert this
inversion of control and use a suspendable I/O procedure,
'read-line/non-blocking', when reading the build log.
* src/cuirass/base.scm (read-line/non-blocking, process-build-log)
(build-derivations&): New procedures.
(%newline, build-event-output-port): Remove.
(spawn-builds): Use 'build-derivations&' instead of 'build-derivations'
with 'build-event-output-port'.
| Ludovic Courtès | 2018-01-29 | 1 | -64/+89 |
