auto merge of #10742 : alexcrichton/rust/frameworks, r=cmr

Commits have the fun details, and scrutiny on the new documentation would be appreciated!

doc/rust.md

@@ -2070,6 +2070,38 @@ The currently implemented features of the compiler are:
                closure as `once` is unlikely to be supported going forward. So
                they are hidden behind this feature until they are to be removed.
 
+* `managed_boxes` - Usage of `@` pointers is gated due to many
+                    planned changes to this feature. In the past, this has meant
+                    "a GC pointer", but the current implementation uses
+                    reference counting and will likely change drastically over
+                    time. Additionally, the `@` syntax will no longer be used to
+                    create GC boxes.
+
+* `asm` - The `asm!` macro provides a means for inline assembly. This is often
+          useful, but the exact syntax for this feature along with its semantics
+          are likely to change, so this macro usage must be opted into.
+
+* `non_ascii_idents` - The compiler supports the use of non-ascii identifiers,
+                       but the implementation is a little rough around the
+                       edges, so this can be seen as an experimental feature for
+                       now until the specification of identifiers is fully
+                       fleshed out.
+
+* `thread_local` - The usage of the `#[thread_local]` attribute is experimental
+                   and should be seen as unstable. This attribute is used to
+                   declare a `static` as being unique per-thread leveraging
+                   LLVM's implementation which works in concert with the kernel
+                   loader and dynamic linker. This is not necessarily available
+                   on all platforms, and usage of it is discouraged (rust
+                   focuses more on task-local data instead of thread-local
+                   data).
+
+* `link_args` - This attribute is used to specify custom flags to the linker,
+                but usage is strongly discouraged. The compiler's usage of the
+                system linker is not guaranteed to continue in the future, and
+                if the system linker is not used then specifying custom flags
+                doesn't have much meaning.
+
 If a feature is promoted to a language feature, then all existing programs will
 start to receive compilation warnings about #[feature] directives which enabled
 the new feature (because the directive is no longer necessary). However, if
@@ -3611,6 +3643,111 @@ queues, as well as code to copy values between queues and their recipients and
 to serialize values for transmission over operating-system inter-process
 communication facilities.
 
+### Linkage
+
+The Rust compiler supports various methods to link crates together both
+statically and dynamically. This section will explore the various methods to
+link Rust crates together, and more information about native libraries can be
+found in the [ffi tutorial][ffi].
+
+In one session of compilation, the compiler can generate multiple artifacts
+through the usage of command line flags and the `crate_type` attribute.
+
+* `--bin`, `#[crate_type = "bin"]` - A runnable executable will be produced.
+  This requires that there is a `main` function in the crate which will be run
+  when the program begins executing. This will link in all Rust and native
+  dependencies, producing a distributable binary.
+
+* `--lib`, `#[crate_type = "lib"]` - A Rust library will be produced. This is
+  an ambiguous concept as to what exactly is produced because a library can
+  manifest itself in several forms. The purpose of this generic `lib` option is
+  to generate the "compiler recommended" style of library. The output library
+  will always be usable by rustc, but the actual type of library may change
+  from time-to-time. The remaining output types are all different flavors of
+  libraries, and the `lib` type can be seen as an alias for one of them (but
+  the actual one is compiler-defined).
+
+* `--dylib`, `#[crate_type = "dylib"]` - A dynamic Rust library will be
+  produced. This is different from the `lib` output type in that this forces
+  dynamic library generation. The resulting dynamic library can be used as a
+  dependency for other libraries and/or executables.  This output type will
+  create `*.so` files on linux, `*.dylib` files on osx, and `*.dll` files on
+  windows.
+
+* `--staticlib`, `#[crate_type = "staticlib"]` - A static system library will
+  be produced. This is different from other library outputs in that the Rust
+  compiler will never attempt to link to `staticlib` outputs. The purpose of
+  this output type is to create a static library containing all of the local
+  crate's code along with all upstream dependencies. The static library is
+  actually a `*.a` archive on linux and osx and a `*.lib` file on windows. This
+  format is recommended for use in situtations such as linking Rust code into an
+  existing non-Rust application because it will not have dynamic dependencies on
+  other Rust code.
+
+* `--rlib`, `#[crate_type = "rlib"]` - A "Rust library" file will be produced.
+  This is used as an intermediate artifact and can be thought of as a "static
+  Rust library". These `rlib` files, unlike `staticlib` files, are interpreted
+  by the Rust compiler in future linkage. This essentially means that `rustc`
+  will look for metadata in `rlib` files like it looks for metadata in dynamic
+  libraries. This form of output is used to produce statically linked
+  executables as well as `staticlib` outputs.
+
+Note that these outputs are stackable in the sense that if multiple are
+specified, then the compiler will produce each form of output at once without
+having to recompile.
+
+With all these different kinds of outputs, if crate A depends on crate B, then
+the compiler could find B in various different forms throughout the system. The
+only forms looked for by the compiler, however, are the `rlib` format and the
+dynamic library format. With these two options for a dependent library, the
+compiler must at some point make a choice between these two formats. With this
+in mind, the compiler follows these rules when determining what format of
+dependencies will be used:
+
+1. If a dynamic library is being produced, then it is required for all upstream
+   Rust dependencies to also be dynamic. This is a limitation of the current
+   implementation of the linkage model.  The reason behind this limitation is to
+   prevent multiple copies of the same upstream library from showing up, and in
+   the future it is planned to support a mixture of dynamic and static linking.
+
+   When producing a dynamic library, the compiler will generate an error if an
+   upstream dependency could not be found, and also if an upstream dependency
+   could only be found in an `rlib` format. Remember that `staticlib` formats
+   are always ignored by `rustc` for crate-linking purposes.
+
+2. If a static library is being produced, all upstream dependecies are
+   required to be available in `rlib` formats. This requirement stems from the
+   same reasons that a dynamic library must have all dynamic dependencies.
+
+   Note that it is impossible to link in native dynamic dependencies to a static
+   library, and in this case warnings will be printed about all unlinked native
+   dynamic dependencies.
+
+3. If an `rlib` file is being produced, then there are no restrictions on what
+   format the upstream dependencies are available in. It is simply required that
+   all upstream dependencies be available for reading metadata from.
+
+   The reason for this is that `rlib` files do not contain any of their upstream
+   dependencies. It wouldn't be very efficient for all `rlib` files to contain a
+   copy of `libstd.rlib`!
+
+4. If an executable is being produced, then things get a little interesting. As
+   with the above limitations in dynamic and static libraries, it is required
+   for all upstream dependencies to be in the same format. The next question is
+   whether to prefer a dynamic or a static format. The compiler currently favors
+   static linking over dynamic linking, but this can be inverted with the `-Z
+   prefer-dynamic` flag to the compiler.
+
+   What this means is that first the compiler will attempt to find all upstream
+   dependencies as `rlib` files, and if successful, it will create a statically
+   linked executable. If an upstream dependency is missing as an `rlib` file,
+   then the compiler will force all dependencies to be dynamic and will generate
+   errors if dynamic versions could not be found.
+
+In general, `--bin` or `--lib` should be sufficient for all compilation needs,
+and the other options are just available if more fine-grained control is desired
+over the output format of a Rust crate.
+
 ### Logging system
 
 The runtime contains a system for directing [logging
@@ -3762,3 +3899,5 @@ Additional specific influences can be seen from the following languages:
 * The typeclass system of Haskell.
 * The lexical identifier rule of Python.
 * The block syntax of Ruby.
+
+[ffi]: tutorial-ffi.html

doc/tutorial-ffi.md

@@ -8,13 +8,13 @@ foreign code. Rust is currently unable to call directly into a C++ library, but
 snappy includes a C interface (documented in
 [`snappy-c.h`](https://code.google.com/p/snappy/source/browse/trunk/snappy-c.h)).
 
-The following is a minimal example of calling a foreign function which will compile if snappy is
+The following is a minimal example of calling a foreign function which will
-installed:
+compile if snappy is installed:
 
 ~~~~ {.xfail-test}
 use std::libc::size_t;
 
-#[link_args = "-lsnappy"]
+#[link(name = "snappy")]
 extern {
     fn snappy_max_compressed_length(source_length: size_t) -> size_t;
 }
@@ -25,26 +25,28 @@ fn main() {
 }
 ~~~~
 
-The `extern` block is a list of function signatures in a foreign library, in this case with the
+The `extern` block is a list of function signatures in a foreign library, in
-platform's C ABI. The `#[link_args]` attribute is used to instruct the linker to link against the
+this case with the platform's C ABI. The `#[link(...)]` attribute is used to
-snappy library so the symbols are resolved.
+instruct the linker to link against the snappy library so the symbols are
+resolved.
 
-Foreign functions are assumed to be unsafe so calls to them need to be wrapped with `unsafe {}` as a
+Foreign functions are assumed to be unsafe so calls to them need to be wrapped
-promise to the compiler that everything contained within truly is safe. C libraries often expose
+with `unsafe {}` as a promise to the compiler that everything contained within
-interfaces that aren't thread-safe, and almost any function that takes a pointer argument isn't
+truly is safe. C libraries often expose interfaces that aren't thread-safe, and
-valid for all possible inputs since the pointer could be dangling, and raw pointers fall outside of
+almost any function that takes a pointer argument isn't valid for all possible
+inputs since the pointer could be dangling, and raw pointers fall outside of
 Rust's safe memory model.
 
-When declaring the argument types to a foreign function, the Rust compiler will not check if the
+When declaring the argument types to a foreign function, the Rust compiler can
-declaration is correct, so specifying it correctly is part of keeping the binding correct at
+not check if the declaration is correct, so specifying it correctly is part of
-runtime.
+keeping the binding correct at runtime.
 
 The `extern` block can be extended to cover the entire snappy API:
 
 ~~~~ {.xfail-test}
 use std::libc::{c_int, size_t};
 
-#[link_args = "-lsnappy"]
+#[link(name = "snappy")]
 extern {
     fn snappy_compress(input: *u8,
                        input_length: size_t,
@@ -232,9 +234,72 @@ fn main() {
 
 # Linking
 
-In addition to the `#[link_args]` attribute for explicitly passing arguments to the linker, an
+The `link` attribute on `extern` blocks provides the basic building block for
-`extern mod` block will pass `-lmodname` to the linker by default unless it has a `#[nolink]`
+instructing rustc how it will link to native libraries. There are two accepted
-attribute applied.
+forms of the link attribute today:
+
+* `#[link(name = "foo")]`
+* `#[link(name = "foo", kind = "bar")]`
+
+In both of these cases, `foo` is the name of the native library that we're
+linking to, and in the second case `bar` is the type of native library that the
+compiler is linking to. There are currently three known types of native
+libraries:
+
+* Dynamic - `#[link(name = "readline")]
+* Static - `#[link(name = "my_build_dependency", kind = "static")]
+* Frameworks - `#[link(name = "CoreFoundation", kind = "framework")]
+
+Note that frameworks are only available on OSX targets.
+
+The different `kind` values are meant to differentiate how the native library
+participates in linkage. From a linkage perspective, the rust compiler creates
+two flavors of artifacts: partial (rlib/staticlib) and final (dylib/binary).
+Native dynamic libraries and frameworks are propagated to the final artifact
+boundary, while static libraries are not propagated at all.
+
+A few examples of how this model can be used are:
+
+* A native build dependency. Sometimes some C/C++ glue is needed when writing
+  some rust code, but distribution of the C/C++ code in a library format is just
+  a burden. In this case, the code will be archived into `libfoo.a` and then the
+  rust crate would declare a dependency via `#[link(name = "foo", kind =
+  "static")]`.
+
+  Regardless of the flavor of output for the crate, the native static library
+  will be included in the output, meaning that distribution of the native static
+  library is not necessary.
+
+* A normal dynamic dependency. Common system libraries (like `readline`) are
+  available on a large number of systems, and often a static copy of these
+  libraries cannot be found. When this dependency is included in a rust crate,
+  partial targets (like rlibs) will not link to the library, but when the rlib
+  is included in a final target (like a binary), the native library will be
+  linked in.
+
+On OSX, frameworks behave with the same semantics as a dynamic library.
+
+## The `link_args` attribute
+
+There is one other way to tell rustc how to customize linking, and that is via
+the `link_args` attribute. This attribute is applied to `extern` blocks and
+specifies raw flags which need to get passed to the linker when producing an
+artifact. An example usage would be:
+
+~~~ {.xfail-test}
+#[link_args = "-foo -bar -baz"]
+extern {}
+~~~
+
+Note that this feature is currently hidden behind the `feature(link_args)` gate
+because this is not a sanctioned way of performing linking. Right now rustc
+shells out to the system linker, so it makes sense to provide extra command line
+arguments, but this will not always be the case. In the future rustc may use
+LLVM directly to link native libraries in which case `link_args` will have no
+meaning.
+
+It is highly recommended to *not* use this attribute, and rather use the more
+formal `#[link(...)]` attribute on `extern` blocks instead.
 
 # Unsafe blocks
 
@@ -260,7 +325,7 @@ blocks with the `static` keyword:
 ~~~{.xfail-test}
 use std::libc;
 
-#[link_args = "-lreadline"]
+#[link(name = "readline")]
 extern {
     static rl_readline_version: libc::c_int;
 }
@@ -279,7 +344,7 @@ them.
 use std::libc;
 use std::ptr;
 
-#[link_args = "-lreadline"]
+#[link(name = "readline")]
 extern {
     static mut rl_prompt: *libc::c_char;
 }

src/librustc/back/link.rs

@@ -1015,7 +1015,7 @@ fn link_rlib(sess: Session, obj_filename: &Path,
             cstore::NativeStatic => {
                 a.add_native_library(l.as_slice());
             }
-            cstore::NativeUnknown => {}
+            cstore::NativeFramework | cstore::NativeUnknown => {}
         }
     }
     return a;
@@ -1044,8 +1044,13 @@ fn link_staticlib(sess: Session, obj_filename: &Path, out_filename: &Path) {
         };
         a.add_rlib(&p);
         let native_libs = csearch::get_native_libraries(sess.cstore, cnum);
-        for lib in native_libs.iter() {
+        for &(kind, ref lib) in native_libs.iter() {
-            sess.warn(format!("unlinked native library: {}", *lib));
+            let name = match kind {
+                cstore::NativeStatic => "static library",
+                cstore::NativeUnknown => "library",
+                cstore::NativeFramework => "framework",
+            };
+            sess.warn(format!("unlinked native {}: {}", name, *lib));
         }
     }
 }
@@ -1204,8 +1209,17 @@ fn add_upstream_rust_crates(args: &mut ~[~str], sess: Session,
                 args.push(cratepath.as_str().unwrap().to_owned());
 
                 let libs = csearch::get_native_libraries(sess.cstore, cnum);
-                for lib in libs.iter() {
+                for &(kind, ref lib) in libs.iter() {
-                    args.push("-l" + *lib);
+                    match kind {
+                        cstore::NativeUnknown => args.push("-l" + *lib),
+                        cstore::NativeFramework => {
+                            args.push(~"-framework");
+                            args.push(lib.to_owned());
+                        }
+                        cstore::NativeStatic => {
+                            sess.bug("statics shouldn't be propagated");
+                        }
+                    }
                 }
             }
             return;
@@ -1262,7 +1276,15 @@ fn add_local_native_libraries(args: &mut ~[~str], sess: Session) {
         args.push("-L" + path.as_str().unwrap().to_owned());
     }
 
-    for &(ref l, _) in cstore::get_used_libraries(sess.cstore).iter() {
+    for &(ref l, kind) in cstore::get_used_libraries(sess.cstore).iter() {
-        args.push(~"-l" + *l);
+        match kind {
+            cstore::NativeUnknown | cstore::NativeStatic => {
+                args.push("-l" + *l);
+            }
+            cstore::NativeFramework => {
+                args.push(~"-framework");
+                args.push(l.to_owned());
+            }
+        }
     }
 }

src/librustc/metadata/common.rs

@@ -199,6 +199,8 @@ pub static tag_region_param_def_def_id: uint = 0x102;
 
 pub static tag_native_libraries: uint = 0x103;
 pub static tag_native_libraries_lib: uint = 0x104;
+pub static tag_native_libraries_name: uint = 0x105;
+pub static tag_native_libraries_kind: uint = 0x106;
 
 pub struct LinkMeta {
     name: @str,

src/librustc/metadata/creader.rs

@@ -18,6 +18,7 @@ use metadata::loader;
 use std::hashmap::HashMap;
 use syntax::ast;
 use std::vec;
+use syntax::abi;
 use syntax::attr;
 use syntax::attr::AttrMetaMethods;
 use syntax::codemap::{Span, dummy_sp};
@@ -191,10 +192,22 @@ fn visit_item(e: &Env, i: @ast::item) {
                             "kind" == k.name()
                         }).and_then(|a| a.value_str());
                         let kind = match kind {
-                            Some(k) if "static" == k => cstore::NativeStatic,
                             Some(k) => {
-                                e.sess.span_fatal(i.span,
+                                if "static" == k {
-                                    format!("unknown kind: `{}`", k));
+                                    cstore::NativeStatic
+                                } else if e.sess.targ_cfg.os == abi::OsMacos &&
+                                          "framework" == k {
+                                    cstore::NativeFramework
+                                } else if "framework" == k {
+                                    e.sess.span_err(m.span,
+                                        "native frameworks are only available \
+                                         on OSX targets");
+                                    cstore::NativeUnknown
+                                } else {
+                                    e.sess.span_err(m.span,
+                                        format!("unknown kind: `{}`", k));
+                                    cstore::NativeUnknown
+                                }
                             }
                             None => cstore::NativeUnknown
                         };
@@ -204,9 +217,10 @@ fn visit_item(e: &Env, i: @ast::item) {
                         let n = match n {
                             Some(n) => n,
                             None => {
-                                e.sess.span_fatal(i.span,
+                                e.sess.span_err(m.span,
                                     "#[link(...)] specified without \
                                      `name = \"foo\"`");
+                                @"foo"
                             }
                         };
                         cstore::add_used_library(cstore, n.to_owned(), kind);

src/librustc/metadata/csearch.rs

@@ -263,7 +263,8 @@ pub fn get_item_visibility(cstore: @mut cstore::CStore,
 }
 
 pub fn get_native_libraries(cstore: @mut cstore::CStore,
-                            crate_num: ast::CrateNum) -> ~[~str] {
+                            crate_num: ast::CrateNum)
+                                -> ~[(cstore::NativeLibaryKind, ~str)] {
     let cdata = cstore::get_crate_data(cstore, crate_num);
     decoder::get_native_libraries(cdata)
 }

src/librustc/metadata/cstore.rs

@@ -40,10 +40,11 @@ pub enum LinkagePreference {
     RequireStatic,
 }
 
-#[deriving(Eq)]
+#[deriving(Eq, FromPrimitive)]
 pub enum NativeLibaryKind {
-    NativeStatic,
+    NativeStatic,    // native static library (.a archive)
-    NativeUnknown,
+    NativeFramework, // OSX-specific
+    NativeUnknown,   // default way to specify a dynamic library
 }
 
 // Where a crate came from on the local filesystem. One of these two options

src/librustc/metadata/decoder.rs

@@ -1530,11 +1530,16 @@ pub fn get_trait_of_method(cdata: Cmd, id: ast::NodeId, tcx: ty::ctxt)
 }
 
 
-pub fn get_native_libraries(cdata: Cmd) -> ~[~str] {
+pub fn get_native_libraries(cdata: Cmd) -> ~[(cstore::NativeLibaryKind, ~str)] {
     let libraries = reader::get_doc(reader::Doc(cdata.data), tag_native_libraries);
     let mut result = ~[];
     reader::tagged_docs(libraries, tag_native_libraries_lib, |lib_doc| {
-        result.push(lib_doc.as_str());
+        let kind_doc = reader::get_doc(lib_doc, tag_native_libraries_kind);
+        let name_doc = reader::get_doc(lib_doc, tag_native_libraries_name);
+        let kind: cstore::NativeLibaryKind =
+            FromPrimitive::from_u32(reader::doc_as_u32(kind_doc)).unwrap();
+        let name = name_doc.as_str();
+        result.push((kind, name));
         true
     });
     return result;

src/librustc/metadata/encoder.rs

@@ -1640,10 +1640,18 @@ fn encode_native_libraries(ecx: &EncodeContext, ebml_w: &mut writer::Encoder) {
     for &(ref lib, kind) in cstore::get_used_libraries(ecx.cstore).iter() {
         match kind {
             cstore::NativeStatic => {} // these libraries are not propagated
-            cstore::NativeUnknown => {
+            cstore::NativeFramework | cstore::NativeUnknown => {
                 ebml_w.start_tag(tag_native_libraries_lib);
+
+                ebml_w.start_tag(tag_native_libraries_kind);
+                ebml_w.writer.write_be_u32(kind as u32);
+                ebml_w.end_tag();
+
+                ebml_w.start_tag(tag_native_libraries_name);
                 ebml_w.writer.write(lib.as_bytes());
                 ebml_w.end_tag();
+
+                ebml_w.end_tag();
             }
         }
     }

src/test/compile-fail/bad-extern-link-attrs.rs

@@ -0,0 +1,16 @@
+// Copyright 2013 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+#[link()] //~ ERROR: specified without `name =
+#[link(name = "foo")]
+#[link(name = "foo", kind = "bar")] //~ ERROR: unknown kind
+extern {}
+
+fn main() {}

src/test/compile-fail/osx-frameworks.rs

@@ -0,0 +1,18 @@
+// Copyright 2013 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// xfail-macos this is supposed to succeed on osx
+
+#[link(name = "foo", kind = "framework")]
+extern {}
+//~^^ ERROR: native frameworks are only available on OSX
+
+fn main() {
+}

src/test/run-pass/osx-frameworks.rs

@@ -0,0 +1,25 @@
+// Copyright 2013 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+use std::libc;
+
+#[cfg(target_os = "macos")]
+#[link(name = "CoreFoundation", kind = "framework")]
+extern {
+    fn CFRunLoopGetTypeID() -> libc::c_ulong;
+}
+
+#[cfg(target_os = "macos")]
+fn main() {
+    unsafe { CFRunLoopGetTypeID(); }
+}
+
+#[cfg(not(target_os = "macos"))]
+pub fn main() {}