Reflection in game engine (c++)

[Gabriel Sassone]

Hello gents,
I wanted to experiment with reflection and I found different way to
obtain it:

1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a
tool that generates headers and cpps

I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out structures,
access fields and change values, stream in/out stuff from network ,and
maybe link with scripting.

My goal is to have fast iteration times, both artists and programmers, and
I achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.

Here are my thoughts, but I would like to hear your ideas and experiences!

1) Very precise on what you can register/serialize, but painful to maintain
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing?
Slow your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.

What are your experiences/thoughts?

Thanks to everyone that read this mail!

Gabriel

[Tinco Andringa]

Hi Gabriel,

In an effort to make C/C++ more like C# I did a bit of research into
adding type/signature information to binaries using clang. A downside
of this approach is that you're restricted to the clang compiler, but
in return you get a lot of power over the compilation process.

The idea is rather simple, the compiler visits all nodes in the C++
AST, you can write a clang module that hooks into that process and
record type information and then at the end store it in a section of
the binary. Then in your code you can simply refer to that section and
read it to get type information.

But that being said, it's a rather ambitious project and I don't
really see how going on such a quest is a wise time investment for
your game pipeline.

Some of the things you mention (serializing, streaming) can be done by
having a neat data model and a good serialization protocol. For
example if your game uses a CES architecture you just need a
serialization for the data structure of each component. Together with
the id's of the components and the objects you could stream in or out
any aspect of the game without ever needing reflection.

I imagine you could even dynamically load/replace components sent over
the network, enabling some sort of hot code swapping :)

Is your game architected in such a way that that would be possible?
Perhaps if your codebase is very large already it's not feasible to
rearchitect to such an extent, though I've heard some studios switched
to CES in the middle of their dev process. It is possible to gradually
migrate from hierarchical architecture to CES (I'm doing so for my own
project atm too).

Hope this is useful.

Kind regards,
Tinco

Post by Gabriel Sassone
Hello gents,
I wanted to experiment with reflection and I found different way to
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a tool
that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out structures,
access fields and change values, stream in/out stuff from network ,and maybe
link with scripting.
My goal is to have fast iteration times, both artists and programmers, and I
achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.
Here are my thoughts, but I would like to hear your ideas and experiences!
1) Very precise on what you can register/serialize, but painful to maintain
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing? Slow
your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Thatcher Ulrich]

"CES"?

Post by Gabriel Sassone
Hello gents,
I wanted to experiment with reflection and I found different way to
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a tool
that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out structures,
access fields and change values, stream in/out stuff from network ,and maybe
link with scripting.
My goal is to have fast iteration times, both artists and programmers, and I
achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.
Here are my thoughts, but I would like to hear your ideas and experiences!
1) Very precise on what you can register/serialize, but painful to maintain
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing? Slow
your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Tinco Andringa]

Sorry, Component Entity System, it's a way of organizing your game
engine that has some popularity amongst in the industry. Here's a
quick overview from someone who worked on the Tony Hawk games:
http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy/

I just read Gabriel's linkedin and he's got years of experience on big
games so I guess he really has a good reason to check if reflection
can optimize his pipeline.

Post by Thatcher Ulrich
"CES"?

Post by Gabriel Sassone
Hello gents,
I wanted to experiment with reflection and I found different way to
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a tool
that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out structures,
access fields and change values, stream in/out stuff from network ,and maybe
link with scripting.
My goal is to have fast iteration times, both artists and programmers, and I
achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.
Here are my thoughts, but I would like to hear your ideas and experiences!
1) Very precise on what you can register/serialize, but painful to maintain
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing? Slow
your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Darren Grant]

Just a cautionary tale:

The trap I've fallen into in the past is not having the tools to really
make the underlying system shine; results vary greatly depending on which
team and resources you have at your disposal. For instance, when designing
a reflective CES it is challenging to come up with a dialect that allows
closely related concepts to map closely together in code, and this has an
impact on the ability to debug and maintain the product later that is
proportional to the complexity of the component ecosystem.

For what it's worth, this is more of a hot topic with language design in
general.


Cheers,
Darren

Post by Tinco Andringa
Sorry, Component Entity System, it's a way of organizing your game
engine that has some popularity amongst in the industry. Here's a
http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy/
I just read Gabriel's linkedin and he's got years of experience on big
games so I guess he really has a good reason to check if reflection
can optimize his pipeline.

Post by Thatcher Ulrich
"CES"?

Post by Gabriel Sassone
Hello gents,
I wanted to experiment with reflection and I found different way to
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you

want/need

Post by Thatcher Ulrich

Post by Gabriel Sassone
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a

tool

Post by Thatcher Ulrich

Post by Gabriel Sassone
that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if

there

Post by Thatcher Ulrich

Post by Gabriel Sassone
are other solutions.
I love the idea to have the possibility to serialize in and out

structures,

Post by Thatcher Ulrich

Post by Gabriel Sassone
access fields and change values, stream in/out stuff from network ,and

maybe

Post by Thatcher Ulrich

Post by Gabriel Sassone
link with scripting.
My goal is to have fast iteration times, both artists and programmers,

and I

Post by Thatcher Ulrich

Post by Gabriel Sassone
achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not

easy to

Post by Thatcher Ulrich

Post by Gabriel Sassone
achieve.
Here are my thoughts, but I would like to hear your ideas and

experiences!

Post by Thatcher Ulrich

Post by Gabriel Sassone
1) Very precise on what you can register/serialize, but painful to

maintain

Post by Thatcher Ulrich

Post by Gabriel Sassone
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can

use to

Post by Thatcher Ulrich

Post by Gabriel Sassone
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations

you

Post by Thatcher Ulrich

Post by Gabriel Sassone
need in your format. Easy to maintain, to plug...maybe slow in parsing?

Slow

Post by Thatcher Ulrich

Post by Gabriel Sassone
your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like

4,

Post by Thatcher Ulrich

Post by Gabriel Sassone
so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

Post by Thatcher Ulrich
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Mike Acton]

*5) create serializable/reflected classes in a data format, then run a tool
that generates headers and cpps*

I'm mostly a fan of #5 (...and don't think it that crazy.) Sandbox off the
problem and solve it as it's own problem rather than deeply mixed into
other problems (e.g. code building)

Our solution shares a lot of similarities with protocol buffers (
https://code.google.com/p/protobuf/) as you'd probably expect.
DDL that generates a JSON schema (for UI mostly) and code (for concrete
structures in-game and tools) that does serialization and all that stuff.
We have a variety of different compilers for different things we might want
to generate. You can check out the core of it here if you're interested:
https://github.com/macton/DDLParser/

Mike.

[Will Vale]

Looks interesting, I like the decoupling that approach offers.

How do you manage data format updates and builds in that scenario? Are the
DDL files something the coding team owns and updates, or are they shared
with (technical) artists, designers etc.?

Will

Post by Mike Acton
*5) create serializable/reflected classes in a data format, then run a
tool that generates headers and cpps*
I'm mostly a fan of #5 (...and don't think it that crazy.) Sandbox off the
problem and solve it as it's own problem rather than deeply mixed into
other problems (e.g. code building)
Our solution shares a lot of similarities with protocol buffers (
https://code.google.com/p/protobuf/) as you'd probably expect.
DDL that generates a JSON schema (for UI mostly) and code (for concrete
structures in-game and tools) that does serialization and all that stuff.
We have a variety of different compilers for different things we might want
https://github.com/macton/DDLParser/
Mike.
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Xpol Wan]

To avoid 1)...5) the are really hard for me.
I prefer create mixed language game engine.
C++ for drawing and low level stuffs
and Script language (eg. Lua) as the interface to game.


Best Regards!

Xpol Wan
// There is a better way!

Post by Will Vale
Looks interesting, I like the decoupling that approach offers.
How do you manage data format updates and builds in that scenario? Are the
DDL files something the coding team owns and updates, or are they shared
with (technical) artists, designers etc.?
Will

Post by Mike Acton
*5) create serializable/reflected classes in a data format, then run a
tool that generates headers and cpps*
I'm mostly a fan of #5 (...and don't think it that crazy.) Sandbox off
the problem and solve it as it's own problem rather than deeply mixed into
other problems (e.g. code building)
Our solution shares a lot of similarities with protocol buffers (
https://code.google.com/p/protobuf/) as you'd probably expect.
DDL that generates a JSON schema (for UI mostly) and code (for concrete
structures in-game and tools) that does serialization and all that stuff.
We have a variety of different compilers for different things we might want
https://github.com/macton/DDLParser/
Mike.
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Mike Acton]

*I prefer create mixed language game engine.*

I'm not sure how that avoids the question? For instance, something still
needs to describe the data exchange between the scripted language and the
native runtime. e.g. We use DDL to generate APIs and data formats for that
kind of thing too. For instance we have a "REST compiler" which generates
an http rest(-ish) interface, with all the data descriptions for our
asset/tools server. It's also used to describe the interfaces between
game-scripting and native code.

[Xpol Wan]

I mean data are defined in Lua.
C++ just do the low level things such as drawing, event etc.
For low level data such bitmap, c++ define it and have a Lua wrapper.
I think we don't need reflection for bitmap object.


Best Regards!

Xpol Wan
// There is a better way!

Post by Mike Acton
*I prefer create mixed language game engine.*
I'm not sure how that avoids the question? For instance, something still
needs to describe the data exchange between the scripted language and the
native runtime. e.g. We use DDL to generate APIs and data formats for that
kind of thing too. For instance we have a "REST compiler" which generates
an http rest(-ish) interface, with all the data descriptions for our
asset/tools server. It's also used to describe the interfaces between
game-scripting and native code.
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Mike Acton]

Hey Will,

*How do you manage data format updates and builds in that scenario? *

Managed in the serialization. It's mostly backward compatible. The source
data is usually JSON formatted. (DDL describes the format that JSON will be
in.) For binary data, we have another scheme. For some DDL we support
multiple formats (JSON, BSON, custom binary) depending on the context. We
also have scripts that manage the version-to-version upgrades (all assets
are in DDL-described formats).

*Are the DDL files something the coding team owns and updates, or are they
shared with (technical) artists, designers etc.?*

Generally speaking, programmers write DDL files. (although programmers in
different groups for different reasons) - usually to map to some concrete
code structure. If you're a non-programmer (let's say you're want to make a
custom tool or something), you'd probably just use an ad hoc JSON format
and not need a formal description. That said, recently we've starting to
have some tech art tools that interface with this stuff so we'll see.

Mike.

[Alen Ladavac]

_______________________________________________
Sweng-Gamedev mailing list
Sweng-***@lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Adrian Stone]

I'm not sure which number describes our approach, but we've been using it
for about 8 years on many different projects and I believe everyone on the
team has been very happy with it. I've written about it pretty extensively
here: http://gameangst.com/?p=107



There are some implementation details I'd change if I were starting over
from scratch, but I still think that using advanced C++ features to create
an embedded DSL is the best option for reflection in C++.



Adrian





_____

From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Alen
Ladavac
Sent: Thursday, March 14, 2013 6:45 PM
To: Gabriel Sassone
Cc: sweng-***@lists.midnightryder.com; sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)



We've been using #4 in our projects for a decade already and it works a
treat. And... why couldn't you invoke methods? :)

That method is the only really fully portable (AFAIK), as it doesn't really
care what compiler/toolset you have. It is also very light in terms of
runtime requirements, and very easy to use (not much extra typing for each
class, member or function).

Though it is rather involved to implement.

JM2C,
Alen

Thursday, March 14, 2013, 7:09:30 PM, you wrote:




Hello gents,
I wanted to experiment with reflection and I found different way to
obtain it:

1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a tool
that generates headers and cpps

I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out structures,
access fields and change values, stream in/out stuff from network ,and maybe
link with scripting.

My goal is to have fast iteration times, both artists and programmers, and I
achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.

Here are my thoughts, but I would like to hear your ideas and experiences!

1) Very precise on what you can register/serialize, but painful to maintain
or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to
serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing? Slow
your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.

What are your experiences/thoughts?

Thanks to everyone that read this mail!

Gabriel

[Nathan Martz]

We use something very similar to what Adrian describes on his blog here
at DF. The syntax is a bit different, but I'd definitely vouch taking an
approach like the one Adrian describes. One really great upside of this
type of solution is that done right, there's not much boilerplate at all
and you get the benefit of the entire system being implemented in code
that you own and know. Building solutions off of heavy weight tools like
CLANG or a big lexer or whatever may open up big long term opportunities
and eventually minimize the amount of "boilerplate" someone needs to
write, but it requires you (and anyone who will extend or maintain the
system) to develop deep knowledge of some seriously complex software, so
much so that it may make the investment prohibitive.



-Nathan



From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of
Adrian Stone
Sent: Thursday, March 14, 2013 4:29 PM
To: sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)





I'm not sure which number describes our approach, but we've been using
it for about 8 years on many different projects and I believe everyone
on the team has been very happy with it. I've written about it pretty
extensively here: http://gameangst.com/?p=107



There are some implementation details I'd change if I were starting over
from scratch, but I still think that using advanced C++ features to
create an embedded DSL is the best option for reflection in C++.



Adrian





________________________________

From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Alen
Ladavac
Sent: Thursday, March 14, 2013 6:45 PM
To: Gabriel Sassone
Cc: sweng-***@lists.midnightryder.com;
sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)



We've been using #4 in our projects for a decade already and it works a
treat. And... why couldn't you invoke methods? :)

That method is the only really fully portable (AFAIK), as it doesn't
really care what compiler/toolset you have. It is also very light in
terms of runtime requirements, and very easy to use (not much extra
typing for each class, member or function).

Though it is rather involved to implement.

JM2C,
Alen

Thursday, March 14, 2013, 7:09:30 PM, you wrote:



Hello gents,
I wanted to experiment with reflection and I found different way to
obtain it:

1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you
want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a
tool that generates headers and cpps

I am trying to figure out flaws and merits of each kind, and even if
there are other solutions.
I love the idea to have the possibility to serialize in and out
structures, access fields and change values, stream in/out stuff from
network ,and maybe link with scripting.

My goal is to have fast iteration times, both artists and programmers,
and I achieved it for rendering programmers already with having json
binarized configurable rendering, but still there are code stuff that
are not easy to achieve.

Here are my thoughts, but I would like to hear your ideas and
experiences!

1) Very precise on what you can register/serialize, but painful to
maintain or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use
to serialize/reflect. Still bad in maintaining, but better in plugging
in.
3) You make the compiler do the work, then translate the informations
you need in your format. Easy to maintain, to plug...maybe slow in
parsing? Slow your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like
4, so data reflection, but you cannot invoke methods if you want.

What are your experiences/thoughts?

Thanks to everyone that read this mail!

Gabriel

[Will Vale]

I've tried a few versions of #1 over the years, and the thing that always
bugged me was repetition - I really don't like it when you have to
duplicate types in the member definitions. I also (having tried it) wanted
to avoid deriving from a common base class to get reflection features. And
finally I didn't want to use templates to actually work with reflected
objects - I'd rather sacrifice a bit of compile-time proveability for not
generating N versions of things.

My current take on this has the usual DECLARE macro in the header for
compound types:

struct Ingredient{ SIL_RTTI_COMPOUND(Ingredient, void) // No
base union { /// Type of the facet,
stored as a hash Nat hash; /// System
implementing this facet mutable ISystem *system;
} type; /// Constant data for the facet, which can be re-used
by different entities const void *data; /// Where to
place the facet w.r.t. the entity Location offset; ///
Anchor facet indices Nat8 anchors[MAX_ANCHORS];};


and looks like this in the source file:

SIL_RTTI(sil::game::Ingredient, SIL_MEMBER(type.hash)
SIL_MEMBER(data) SIL_MEMBER(offset) SIL_MEMBER(anchors))


(Templates deduce types for members.)


This generates an instance of a class derived from rtti::IType, of
which there are a small hierarchy representing different kinds of C++
type - atoms, compounds, pointers, C arrays etc. When you work with
arbitrary reflected instances you get a void pointer and an IType
reference, and write bespoke code to walk the types and members to
achieve something. It usually looks like this so it's not quite as
scary as void* would suggest:


// Generic code

void really_do_stuff(const rtti::IType& type, void *instance)

{

// Inspect type and do things with *instance

}


// Type-safe layer which gets compiled away - Type<T>() (usually) gets
compiled to "load address of type data"

template <typename T> void do_stuff(T& object)

{

really_do_stuff(rtti::Type<T>(), &object);

}


The code you write in really_do_stuff can get a bit boilerplate-ey,
but you don't have to do it often, and there are ways of reducing the
boilerplate (iterator, visitor) which can be layered on top. I've
generally found it easier to write the code out rather than coerce it
into the iterator's view though. Here's a (small extract) of real code
which constructs objects from text definitions:


static void build_compound( const PropertyTree::Group& group, const
rtti::ICompoundType &type, void *instance, Librarian& librarian )

{ for ( const rtti::ICompoundType *t = &type; t; t= t->Base() )
{ for ( Nat m = 0; m != t->MemberCount(); ++m )
{ const
rtti::ICompoundType::Member& member = t->GetMember(m);
// Lookup once (smaller code, more wasteful runtime)
const PropertyTree::Group* member_group =
group.FindGroup(member.Name()); const
PropertyTree::Property *member_prop =
group.FindProperty(member.Name()); // Handle
member's type build_instance(member_group,
member_prop, member.Type(), OffsetPtr(instance, member.offset),
librarian); } instance =
OffsetPtr(instance, t->BaseOffset()); }}


This kind of thing doesn't get used at runtime, but it's used in tools
to build "memory image"-style data files which are used at runtime.
Loading these uses a pinch of reflection for calling constructors etc,
but it's not much.


The nice thing about this (and other variations of #1) is that if it
ends up not working you can switch to #4 without re-annotating your
source :)


Will

We use something very similar to what Adrian describes on his blog here at
DF. The syntax is a bit different, but I’d definitely vouch taking an
approach like the one Adrian describes. One really great upside of this
type of solution is that done right, there’s not much boilerplate at all
and you get the benefit of the entire system being implemented in code that
you own and know. Building solutions off of heavy weight tools like CLANG
or a big lexer or whatever may open up big long term opportunities and
eventually minimize the amount of “boilerplate” someone needs to write, but
it requires you (and anyone who will extend or maintain the system) to
develop deep knowledge of some seriously complex software, so much so that
it may make the investment prohibitive.****
** **
-Nathan****
** **
*Sent:* Thursday, March 14, 2013 4:29 PM
*Subject:* Re: [Sweng-Gamedev] Reflection in game engine (c++)****
** **
** **
I’m not sure which number describes our approach, but we’ve been using it
for about 8 years on many different projects and I believe everyone on the
team has been very happy with it. I’ve written about it pretty extensively
here: http://gameangst.com/?p=107****
** **
There are some implementation details I’d change if I were starting over
from scratch, but I still think that using advanced C++ features to create
an embedded DSL is the best option for reflection in C++.****
** **
Adrian****
** **
** **
------------------------------
*On Behalf Of *Alen Ladavac
*Sent:* Thursday, March 14, 2013 6:45 PM
*To:* Gabriel Sassone
*Subject:* Re: [Sweng-Gamedev] Reflection in game engine (c++)****
** **
We've been using #4 in our projects for a decade already and it works a
treat. And... why couldn't you invoke methods? :)
That method is the only really fully portable (AFAIK), as it doesn't
really care what compiler/toolset you have. It is also very light in terms
of runtime requirements, and very easy to use (not much extra typing for
each class, member or function).
Though it is rather involved to implement.
JM2C,
Alen
Thursday, March 14, 2013, 7:09:30 PM, you wrote:****
** **
Hello gents,
I wanted to experiment with reflection and I found different way to
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations
(like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a
tool that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if there
are other solutions.
I love the idea to have the possibility to serialize in and out
structures, access fields and change values, stream in/out stuff from
network ,and maybe link with scripting.
My goal is to have fast iteration times, both artists and programmers, and
I achieved it for rendering programmers already with having json binarized
configurable rendering, but still there are code stuff that are not easy to
achieve.
Here are my thoughts, but I would like to hear your ideas and experiences!
1) Very precise on what you can register/serialize, but painful to
maintain or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use
to serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you
need in your format. Easy to maintain, to plug...maybe slow in parsing?
Slow your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke
methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4,
so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel****
*--
Best regards,
****
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Jarkko Lempiainen]

Not a big fan of #5 I have used in few projects or reflection solutions in general which require custom tools/languages in your build system. I have opted for defining reflection in C++ with macros. There are few benefits:
- Can write any reflection logic since the refl. definition is essentially C++ code.
- All types use the same system (e.g. no special treatment of lightweight POD types needed)
- Easy to add reflection to classes as an afterthought without need to convert to a custom language
- Non-intrusive to your build system and no external tools slowing the build process
- Supports proper encapsulation
- Surpports data driven dynamic reflection, i.e. different class instances may expose different set of members based on data
- Easily debuggable
- Cross-platform

Downsides:
- Impacts your compilation time
- Not as neat syntax as custom language
- Need to repeat variable names for reflection

Cheers, Jarkko

Post by Gabriel Sassone
Hello gents,
1) Invasive macro based registration
2) Reflection visitor pattern
3) Parsing pdb/clang stuff and create a manual table of what you want/need
4) c++ custom parser that uses tags to create reflection informations (like a comment near a member, ...) (exuberant ctags???)
5) create serializable/reflected classes in a data format, then run a tool that generates headers and cpps
I am trying to figure out flaws and merits of each kind, and even if there are other solutions.
I love the idea to have the possibility to serialize in and out structures, access fields and change values, stream in/out stuff from network ,and maybe link with scripting.
My goal is to have fast iteration times, both artists and programmers, and I achieved it for rendering programmers already with having json binarized configurable rendering, but still there are code stuff that are not easy to achieve.
Here are my thoughts, but I would like to hear your ideas and experiences!
1) Very precise on what you can register/serialize, but painful to maintain or plug into an existing codebase
2) Less precise but you just need one method per class, that you can use to serialize/reflect. Still bad in maintaining, but better in plugging in.
3) You make the compiler do the work, then translate the informations you need in your format. Easy to maintain, to plug...maybe slow in parsing? Slow your build pipeline?
4) maybe easier than using pdbs, but less powerful (you cannot invoke methods on objects). should be faster than 3.
5) crazy idea. requires generation of code from outside...powerful like 4, so data reflection, but you cannot invoke methods if you want.
What are your experiences/thoughts?
Thanks to everyone that read this mail!
Gabriel
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Massimo Del Zotto]

I have limited experience but I have been very impressed by the separation
provided by scripting languages.
Considering the goals:

*My goal is to have fast iteration times, both artists and programmers*

It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.

By the way, *I would like to have an insight on how this reflection
information is to be used*. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.

Massimo

[Don Williamson]

You will never get an ideal C++ reflection system; you just have to pick
one technique that:

* Is as simple and transparent as possible.
* Suits the programming background of the majority of your team.
* Gives most benefits for the target game (not tech for tech's sake).

All methods can achieve that, but all methods can similarly bring your
entire code base to an over-engineered death of spectacular proportions. Be
careful and don't try to solve too many problems that C++ wasn't designed
to solve in the first place.

With that in mind, here's some experiences I've shared in the past...

This post covers a broad overview of various reflection techniques:

http://www.altdevblogaday.com/2011/09/25/reflection-in-c-part-1-introduction/

This post covers the SC5 approach, which includes live C++ code reloading
for the engine (we were doing this back in 2005):

http://www.altdevblogaday.com/2012/01/03/reflection-in-c-part-2-the-simple-implementation-of-splinter-cell/

I was working on part 3 that covers the use of IDL or DDL files. I can't
seem to find the latest draft (I've gotten really lazy over this) but I
still have a draft upload from a while ago:
http://donw.org/b/?d=adr

My own company's code base is engineered with a clang parser that does all
this automatically. I've open-sourced most of it as clReflect:
https://bitbucket.org/dwilliamson/clreflect

I use it for transparent interop between native C++ and javascript browser
code, live C++ code reloading, serialisation, replay logic, network code
and many other things. It also compiles across Windows, Mac OSX and Linux.
However, it's not seeing many updates these days because it's very stable
for the target application I made it for, meaning it may not work out of
the box for you (patches make me happy!). One nice point, is that you
probably won't see a simpler runtime reflection engine as there is no
registration required at runtime, beyond loading a binary blob and patching
up pointers.

Some alternative research implementations include:
https://bitbucket.org/dwilliamson/reflectabit - Manual registration with
C++ templates
https://bitbucket.org/dwilliamson/rfl - Auto-reflection through PDB
files (very powerful but may break any day MS decides to change or hide
their API).

So, yeah... pick one. But try to make it as invisible to your engineers as
possible :)

Cheers,
- Don

[Tinco Andringa]

I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.

Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Douglas Cox]

That sounds pretty similar to the goals of the engine that we're working on
-- basically faster iteration times for everyone (this includes engine
programmers as well).

All of our code (including the compiler) is written in a C#-like language,
so we have perfect reflection from that. Attributes are pretty much
required for providing additional information to property editors for
things like valid ranges of a field or notifications of a property change
(most of which can go away in Final builds).

Another thing to consider may be allowing for data derivation. This can
save quite a bit of work when the need for copies of objects come up that
only need one or two properties changed. If you pick a serialization
format for the editable data that knows about the data (fieldtype +
fieldname + value), it makes it a lot easier to get this working.

In the past, we've used Macros and they worked fine. They're a bit more
annoying to extend with optional parameters and you have to repeat the
field/property name, but that's not terrible since you can generally get it
down to compile-time errors if you get something wrong (other than
forgetting to add a new field).

-Doug

Post by Tinco Andringa
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the

separation

Post by Massimo Del Zotto
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side

of

Post by Massimo Del Zotto
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use

a

Post by Massimo Del Zotto
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Adrian Stephens]

I'm glad we're still talking about this - I think it's important and I wanted to add my own observations:

The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.

If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.

To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.

This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.

Some other random thoughts:
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)

struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };

template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }



Adrian Stephens

That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Mike Shaver]

When I did this for a non-game app, I defined it with self-aware, strong-AI macros, and compiled the resulting data into a shared library. The app (game) linked against the library, as did one important tool: the exporter. It was a wee little program that dumped the compiled metadata to an interchange format. Today I would choose JSON or XML. The build process would regenerate metadata.json whenever metadata.so changed, and extending it to multiple libraries/output files was straightforward.

Runtime reflection formats have many constraints that tool-interchange formats don't, and tool formats often want things the runtime abhors. Put the line-numbers in the output and remix with git-blame output to tell you who last changed an entity? Sure. Web explorer of the resources, without linking the parser library into PHP...I'll be over here.

I would similarly not use Collada files at runtime.

phone-typed

Post by Adrian Stephens
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens

That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug

Post by Tinco Andringa
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Adrian Stephens]

So you're writing macros to turn your C structures into data, building them into a library, linking it with your exporter and running the exporter to generate a script-format of the structures. Which, then has to be parsed by your tools. I honestly didn't mean that to sound so negative, but wouldn't it have been easier the other way round - start with a simple domain-specific language to define your types and spit out C header files?

The PHP thing works because you're proposing outputting JSON, which somebody presumably has already linked into it. If that sort of thing is difficult you could certainly output JSON at the same time you output C header files, or I suppose you could probably even use JSON *as* your script format.

Post by Mike Shaver
When I did this for a non-game app, I defined it with self-aware, strong-AI macros, and compiled the resulting data into a shared library. The app (game) linked against the library, as did one important tool: the exporter. It was a wee little program that dumped the compiled metadata to an interchange format. Today I would choose JSON or XML. The build process would regenerate metadata.json whenever metadata.so changed, and extending it to multiple libraries/output files was straightforward.
Runtime reflection formats have many constraints that tool-interchange formats don't, and tool formats often want things the runtime abhors. Put the line-numbers in the output and remix with git-blame output to tell you who last changed an entity? Sure. Web explorer of the resources, without linking the parser library into PHP...I'll be over here.
I would similarly not use Collada files at runtime.
phone-typed

Post by Adrian Stephens
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens

That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Mike Shaver]

I wanted to just have one place that things were declared, and for that to be with real code rather in a separate "please don't edit this no matter how much VC++ tells you this is the definition" file.

My point is really that it's over-constraining the problem to choose one format that is both fast+compact enough for runtime use, and flexible+straightforward enough to bend into all the parts of the tool pipeline.

It's the work of 5 lines of Python to go from JSON to XML or CSV or .ini files or whatever. That someone has linked the format parser into a dozen different software environments is indeed an advantage that I would take seriously. I generally don't know what my tools are going to be until I've written them, so free option value is pretty nice.

I've done the "IDL to headers and packed metadata files" thing, and it was pretty annoying (and error-prone) to wire it into even a half-dozen straight-C tools. I may be overreacting to that experience, but mutatis mutandis I very much prefer one tiny exporter and then letting tools transcode however they want.

Mike

phone-typed

Post by Adrian Stephens
So you're writing macros to turn your C structures into data, building them into a library, linking it with your exporter and running the exporter to generate a script-format of the structures. Which, then has to be parsed by your tools. I honestly didn't mean that to sound so negative, but wouldn't it have been easier the other way round - start with a simple domain-specific language to define your types and spit out C header files?
The PHP thing works because you're proposing outputting JSON, which somebody presumably has already linked into it. If that sort of thing is difficult you could certainly output JSON at the same time you output C header files, or I suppose you could probably even use JSON *as* your script format.

Post by Mike Shaver
When I did this for a non-game app, I defined it with self-aware, strong-AI macros, and compiled the resulting data into a shared library. The app (game) linked against the library, as did one important tool: the exporter. It was a wee little program that dumped the compiled metadata to an interchange format. Today I would choose JSON or XML. The build process would regenerate metadata.json whenever metadata.so changed, and extending it to multiple libraries/output files was straightforward.
Runtime reflection formats have many constraints that tool-interchange formats don't, and tool formats often want things the runtime abhors. Put the line-numbers in the output and remix with git-blame output to tell you who last changed an entity? Sure. Web explorer of the resources, without linking the parser library into PHP...I'll be over here.
I would similarly not use Collada files at runtime.
phone-typed

Post by Adrian Stephens
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens

That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug

Post by Tinco Andringa
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Douglas Cox]

One issue that we were thinking about after coming up with a file format
using reflection that allows for data-derivation, versioning, etc., and
that basically looks like "field name + field type + value" (only
overriding/saving values that were changed) was: "Is it worth trying to
optimize or flatten this for final builds?"

At first we were just going to write a different serializer and write out a
flattened set of in-order values thinking that our types would never
change. But with DLC and patches, this is not the case. We may release a
patch that changes a type (adds or removes a field) and existing
(non-patched) data files would fail to load.

Another reason to not flatten is that it is quite possible that the final
data in the 'overrides only' format would be much less storage than the
flattened format. And of course not having to write additional code that's
only run in final builds is a plus.

Anyone else have thoughts on this or do it differently?

--
And as far as the debate over where to put the type info, all I can say is
that having the compiler generate it and having 1 file (a single ".x"
instead of ".h" and ".cpp" and ".inl") for code has been great. I
definitely do not miss bouncing back and forth between .h and .cpp files.
So if you do start wandering into the land of making your own language for
just reflection, you might see what other benefits you can get from doing
more in it than just that.
--

-Doug

Post by Mike Shaver
I wanted to just have one place that things were declared, and for that to
be with real code rather in a separate "please don't edit this no matter
how much VC++ tells you this is the definition" file.
My point is really that it's over-constraining the problem to choose one
format that is both fast+compact enough for runtime use, and
flexible+straightforward enough to bend into all the parts of the tool
pipeline.
It's the work of 5 lines of Python to go from JSON to XML or CSV or .ini
files or whatever. That someone has linked the format parser into a dozen
different software environments is indeed an advantage that I would take
seriously. I generally don't know what my tools are going to be until I've
written them, so free option value is pretty nice.
I've done the "IDL to headers and packed metadata files" thing, and it was
pretty annoying (and error-prone) to wire it into even a half-dozen
straight-C tools. I may be overreacting to that experience, but mutatis
mutandis I very much prefer one tiny exporter and then letting tools
transcode however they want.
Mike
phone-typed
So you're writing macros to turn your C structures into data, building
them into a library, linking it with your exporter and running the exporter
to generate a script-format of the structures. Which, then has to be
parsed by your tools. I honestly didn't mean that to sound so negative,
but wouldn't it have been easier the other way round - start with a simple
domain-specific language to define your types and spit out C header files?
The PHP thing works because you're proposing outputting JSON, which
somebody presumably has already linked into it. If that sort of thing is
difficult you could certainly output JSON at the same time you output C
header files, or I suppose you could probably even use JSON *as* your
script format.
When I did this for a non-game app, I defined it with self-aware,
strong-AI macros, and compiled the resulting data into a shared library.
The app (game) linked against the library, as did one important tool: the
exporter. It was a wee little program that dumped the compiled metadata to
an interchange format. Today I would choose JSON or XML. The build process
would regenerate metadata.json whenever metadata.so changed, and extending
it to multiple libraries/output files was straightforward.
Runtime reflection formats have many constraints that tool-interchange
formats don't, and tool formats often want things the runtime abhors. Put
the line-numbers in the output and remix with git-blame output to tell you
who last changed an entity? Sure. Web explorer of the resources, without
linking the parser library into PHP...I'll be over here.
I would similarly not use Collada files at runtime.
phone-typed
I'm glad we're still talking about this - I think it's important and I
The macro/template approach to reflection only applies to one side of the
data/code interface. If all you need to do is serialize runtime data to be
read back in by runtime code, then you're ok; but if you want to access
that metadata from an external tool, you either have to recompile the tool
or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you
might as well use it for both code and data views. Which means you can use
a more natural syntax for your structure definitions that even
non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out
header files containing typedefs that correspond to the metadata, and
generate whatever you need to allow the runtime to interpret serialized
data. This also address the separation of code and assets: we don't want
artists having to sync the code, and this keeps the data definitions with
the assets that they define.
This is what I do, anyway. I have a data language, which defines types,
and which can also be used to define data, and all data is stored using
these definitions. This includes bitmaps, sounds, models, etc. The data
language is separated from the data representation so this doesn't imply
that bitmaps are stored in a script-like array of colors - but it does mean
that tools can use the reflection system on more classes of data than just
small gameplay entities. And when distinctions like that are removed, and
all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort
of fieldsof(T) operator that enumerates the fields of a structure. e.g.
(typed straight into email, so probably bollocks - and not representative
of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t;
md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens
That sounds pretty similar to the goals of the engine that we're working
on -- basically faster iteration times for everyone (this includes engine
programmers as well).
All of our code (including the compiler) is written in a C#-like language,
so we have perfect reflection from that. Attributes are pretty much
required for providing additional information to property editors for
things like valid ranges of a field or notifications of a property change
(most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can
save quite a bit of work when the need for copies of objects come up that
only need one or two properties changed. If you pick a serialization
format for the editable data that knows about the data (fieldtype +
fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more
annoying to extend with optional parameters and you have to repeat the
field/property name, but that's not terrible since you can generally get it
down to compile-time errors if you get something wrong (other than
forgetting to add a new field).
-Doug

Post by Tinco Andringa
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the

separation

Post by Massimo Del Zotto
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist

side of

Post by Massimo Del Zotto
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd

use a

Post by Massimo Del Zotto
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Adrian Stephens]

We flatten our data, so reading data in-game is just a question of allocating enough space (from the different flavors of memory - normal, video, sound, etc) reading the data and patching the results. During development there is the possibility of reading some out-of-date types, and those need to be converted during the patch-up, which means they do take extra memory. The shipped code shouldn't have to do that.

That system continues to work with dlc and patches, but if your patch needs to change a type, you'd probably be sending new data files using that type anyway.

We usually read these files on a separate thread, and it's comforting not to be making lots of allocs that might fragment memory if interleaved with allocations that might happen on other threads.

Regarding storage space - if you compress your binary files, as we do, any often-used default values are likely to compress away very easily; and since our data is flattened and compressed during development, there's no additional code to worry about.

Adrian

One issue that we were thinking about after coming up with a file format using reflection that allows for data-derivation, versioning, etc., and that basically looks like "field name + field type + value" (only overriding/saving values that were changed) was: "Is it worth trying to optimize or flatten this for final builds?"
At first we were just going to write a different serializer and write out a flattened set of in-order values thinking that our types would never change. But with DLC and patches, this is not the case. We may release a patch that changes a type (adds or removes a field) and existing (non-patched) data files would fail to load.
Another reason to not flatten is that it is quite possible that the final data in the 'overrides only' format would be much less storage than the flattened format. And of course not having to write additional code that's only run in final builds is a plus.
Anyone else have thoughts on this or do it differently?
--
And as far as the debate over where to put the type info, all I can say is that having the compiler generate it and having 1 file (a single ".x" instead of ".h" and ".cpp" and ".inl") for code has been great. I definitely do not miss bouncing back and forth between .h and .cpp files. So if you do start wandering into the land of making your own language for just reflection, you might see what other benefits you can get from doing more in it than just that.
--
-Doug
I wanted to just have one place that things were declared, and for that to be with real code rather in a separate "please don't edit this no matter how much VC++ tells you this is the definition" file.
My point is really that it's over-constraining the problem to choose one format that is both fast+compact enough for runtime use, and flexible+straightforward enough to bend into all the parts of the tool pipeline.
It's the work of 5 lines of Python to go from JSON to XML or CSV or .ini files or whatever. That someone has linked the format parser into a dozen different software environments is indeed an advantage that I would take seriously. I generally don't know what my tools are going to be until I've written them, so free option value is pretty nice.
I've done the "IDL to headers and packed metadata files" thing, and it was pretty annoying (and error-prone) to wire it into even a half-dozen straight-C tools. I may be overreacting to that experience, but mutatis mutandis I very much prefer one tiny exporter and then letting tools transcode however they want.
Mike
phone-typed

Post by Adrian Stephens
So you're writing macros to turn your C structures into data, building them into a library, linking it with your exporter and running the exporter to generate a script-format of the structures. Which, then has to be parsed by your tools. I honestly didn't mean that to sound so negative, but wouldn't it have been easier the other way round - start with a simple domain-specific language to define your types and spit out C header files?
The PHP thing works because you're proposing outputting JSON, which somebody presumably has already linked into it. If that sort of thing is difficult you could certainly output JSON at the same time you output C header files, or I suppose you could probably even use JSON *as* your script format.

Post by Mike Shaver
When I did this for a non-game app, I defined it with self-aware, strong-AI macros, and compiled the resulting data into a shared library. The app (game) linked against the library, as did one important tool: the exporter. It was a wee little program that dumped the compiled metadata to an interchange format. Today I would choose JSON or XML. The build process would regenerate metadata.json whenever metadata.so changed, and extending it to multiple libraries/output files was straightforward.
Runtime reflection formats have many constraints that tool-interchange formats don't, and tool formats often want things the runtime abhors. Put the line-numbers in the output and remix with git-blame output to tell you who last changed an entity? Sure. Web explorer of the resources, without linking the parser library into PHP...I'll be over here.
I would similarly not use Collada files at runtime.
phone-typed

Post by Adrian Stephens
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens

That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Douglas Cox]

"That system continues to work with dlc and patches, but if your patch
needs to change a type, you'd probably be sending new data files using that
type anyway."

You'd think so, but we've had a few examples of this already in the last
few years. Only a few new instances needed to override a default value.

"We usually read these files on a separate thread, and it's comforting not
to be making lots of allocs that might fragment memory if interleaved with
allocations that might happen on other threads."

This seems like it would happen regardless unless your main 'runtime'
allocations are coming from a different memory pool.

I probably should have mentioned that I was also talking about overriding
values on 'prefabs' with this same system. So when we override just a few
position/rotations it is saving considerably more than flattening the
entire prefab into the level file for instance.

Sorry I should have made this a separate thread.

-Doug

Post by Adrian Stephens
We flatten our data, so reading data in-game is just a question of
allocating enough space (from the different flavors of memory - normal,
video, sound, etc) reading the data and patching the results. During
development there is the possibility of reading some out-of-date types, and
those need to be converted during the patch-up, which means they do take
extra memory. The shipped code shouldn't have to do that.
That system continues to work with dlc and patches, but if your patch
needs to change a type, you'd probably be sending new data files using that
type anyway.
We usually read these files on a separate thread, and it's comforting not
to be making lots of allocs that might fragment memory if interleaved with
allocations that might happen on other threads.
Regarding storage space - if you compress your binary files, as we do, any
often-used default values are likely to compress away very easily; and
since our data is flattened and compressed during development, there's no
additional code to worry about.
Adrian
One issue that we were thinking about after coming up with a file format
using reflection that allows for data-derivation, versioning, etc., and
that basically looks like "field name + field type + value" (only
overriding/saving values that were changed) was: "Is it worth trying to
optimize or flatten this for final builds?"
At first we were just going to write a different serializer and write out
a flattened set of in-order values thinking that our types would never
change. But with DLC and patches, this is not the case. We may release a
patch that changes a type (adds or removes a field) and existing
(non-patched) data files would fail to load.
Another reason to not flatten is that it is quite possible that the final
data in the 'overrides only' format would be much less storage than the
flattened format. And of course not having to write additional code that's
only run in final builds is a plus.
Anyone else have thoughts on this or do it differently?
--
And as far as the debate over where to put the type info, all I can say is
that having the compiler generate it and having 1 file (a single ".x"
instead of ".h" and ".cpp" and ".inl") for code has been great. I
definitely do not miss bouncing back and forth between .h and .cpp files.
So if you do start wandering into the land of making your own language for
just reflection, you might see what other benefits you can get from doing
more in it than just that.
--
-Doug

Post by Mike Shaver
I wanted to just have one place that things were declared, and for that
to be with real code rather in a separate "please don't edit this no matter
how much VC++ tells you this is the definition" file.
My point is really that it's over-constraining the problem to choose one
format that is both fast+compact enough for runtime use, and
flexible+straightforward enough to bend into all the parts of the tool
pipeline.
It's the work of 5 lines of Python to go from JSON to XML or CSV or .ini
files or whatever. That someone has linked the format parser into a dozen
different software environments is indeed an advantage that I would take
seriously. I generally don't know what my tools are going to be until I've
written them, so free option value is pretty nice.
I've done the "IDL to headers and packed metadata files" thing, and it
was pretty annoying (and error-prone) to wire it into even a half-dozen
straight-C tools. I may be overreacting to that experience, but mutatis
mutandis I very much prefer one tiny exporter and then letting tools
transcode however they want.
Mike
phone-typed
So you're writing macros to turn your C structures into data, building
them into a library, linking it with your exporter and running the exporter
to generate a script-format of the structures. Which, then has to be
parsed by your tools. I honestly didn't mean that to sound so negative,
but wouldn't it have been easier the other way round - start with a simple
domain-specific language to define your types and spit out C header files?
The PHP thing works because you're proposing outputting JSON, which
somebody presumably has already linked into it. If that sort of thing is
difficult you could certainly output JSON at the same time you output C
header files, or I suppose you could probably even use JSON *as* your
script format.
When I did this for a non-game app, I defined it with self-aware,
strong-AI macros, and compiled the resulting data into a shared library.
The app (game) linked against the library, as did one important tool: the
exporter. It was a wee little program that dumped the compiled metadata to
an interchange format. Today I would choose JSON or XML. The build process
would regenerate metadata.json whenever metadata.so changed, and extending
it to multiple libraries/output files was straightforward.
Runtime reflection formats have many constraints that tool-interchange
formats don't, and tool formats often want things the runtime abhors. Put
the line-numbers in the output and remix with git-blame output to tell you
who last changed an entity? Sure. Web explorer of the resources, without
linking the parser library into PHP...I'll be over here.
I would similarly not use Collada files at runtime.
phone-typed
I'm glad we're still talking about this - I think it's important and I
The macro/template approach to reflection only applies to one side of the
data/code interface. If all you need to do is serialize runtime data to be
read back in by runtime code, then you're ok; but if you want to access
that metadata from an external tool, you either have to recompile the tool
or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you
might as well use it for both code and data views. Which means you can use
a more natural syntax for your structure definitions that even
non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out
header files containing typedefs that correspond to the metadata, and
generate whatever you need to allow the runtime to interpret serialized
data. This also address the separation of code and assets: we don't want
artists having to sync the code, and this keeps the data definitions with
the assets that they define.
This is what I do, anyway. I have a data language, which defines types,
and which can also be used to define data, and all data is stored using
these definitions. This includes bitmaps, sounds, models, etc. The data
language is separated from the data representation so this doesn't imply
that bitmaps are stored in a script-like array of colors - but it does mean
that tools can use the reflection system on more classes of data than just
small gameplay entities. And when distinctions like that are removed, and
all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort
of fieldsof(T) operator that enumerates the fields of a structure. e.g.
(typed straight into email, so probably bollocks - and not representative
of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>,
md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md()
: metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t;
md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens
That sounds pretty similar to the goals of the engine that we're working
on -- basically faster iteration times for everyone (this includes engine
programmers as well).
All of our code (including the compiler) is written in a C#-like
language, so we have perfect reflection from that. Attributes are pretty
much required for providing additional information to property editors for
things like valid ranges of a field or notifications of a property change
(most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can
save quite a bit of work when the need for copies of objects come up that
only need one or two properties changed. If you pick a serialization
format for the editable data that knows about the data (fieldtype +
fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more
annoying to extend with optional parameters and you have to repeat the
field/property name, but that's not terrible since you can generally get it
down to compile-time errors if you get something wrong (other than
forgetting to add a new field).
-Doug

Post by Tinco Andringa
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the

separation

Post by Massimo Del Zotto
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist

side of

Post by Massimo Del Zotto
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd

use a

Post by Massimo Del Zotto
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Jarkko Lempiainen]

If you define your reflection in C++, you can still have external tools for
asset processing, because class definitions must be stored along with
assets. This is something you have to do regardless how you implement your
reflection (assuming you don't have cumbersome "convert all asset to new
version whenever the class definition changes"), because different assets
may use different versions of class definition and you must be able to read
& convert the old data to new version when reading it.



Beyond basic class reflection, the reflection system must also be able to
handle special cases. For example when dealing with textures, you reflection
system must support "custom reflection" definitions where textures are read
straight to the video memory without intermediate data store. Or some
classes may expose different set of members depending on the instance of the
class (think of exposing variants for example). I'm not sure how you would
implement these with reflection defined in script without special hacks in
the reflection system.





Cheers, Jarkko





From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Adrian
Stephens
Sent: Monday, March 18, 2013 7:31 PM
To: sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)





I'm glad we're still talking about this - I think it's important and I
wanted to add my own observations:



The macro/template approach to reflection only applies to one side of the
data/code interface. If all you need to do is serialize runtime data to be
read back in by runtime code, then you're ok; but if you want to access that
metadata from an external tool, you either have to recompile the tool or
plugin when anything changes, or write something that parses that code.



If you're going to have to write a parser anyway, it seems to me you might
as well use it for both code and data views. Which means you can use a more
natural syntax for your structure definitions that even non-programmers can
use.



To map this back into your C++ code, you can pretty trivially write out
header files containing typedefs that correspond to the metadata, and
generate whatever you need to allow the runtime to interpret serialized
data. This also address the separation of code and assets: we don't want
artists having to sync the code, and this keeps the data definitions with
the assets that they define.



This is what I do, anyway. I have a data language, which defines types, and
which can also be used to define data, and all data is stored using these
definitions. This includes bitmaps, sounds, models, etc. The data language
is separated from the data representation so this doesn't imply that bitmaps
are stored in a script-like array of colors - but it does mean that tools
can use the reflection system on more classes of data than just small
gameplay entities. And when distinctions like that are removed, and all
things are treated the same way, lots of clever things just fall out.



Some other random thoughts:

As far as I can see, all C++ needs to allow full reflection is some sort of
fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed
straight into email, so probably bollocks - and not representative of my
actual system!)



struct metadata {

enum TYPE {INT, PTR, ARRAY, STRUCT} type;

metadata(TYPE &_type) : type(_type) {}

};

template<typename A, typename... B> struct md_fields : md<A>, md_fields<B>
{};

template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() :
metadata(STRUCT) {} };

template<typename T> struct md<T*> : metadata { md<T> t; md() :
metadata(PTR) {} };

template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t;
md() : metadata(ARRAY), n(N) {} };

template<> struct md<int> : metadata { md() : metadata(INT) {} };



template<typename T> metadata *get_md(const T &t) { static md<T> m; return
&m; }







Adrian Stephens



On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote:





That sounds pretty similar to the goals of the engine that we're working on
-- basically faster iteration times for everyone (this includes engine
programmers as well).



All of our code (including the compiler) is written in a C#-like language,
so we have perfect reflection from that. Attributes are pretty much
required for providing additional information to property editors for things
like valid ranges of a field or notifications of a property change (most of
which can go away in Final builds).



Another thing to consider may be allowing for data derivation. This can
save quite a bit of work when the need for copies of objects come up that
only need one or two properties changed. If you pick a serialization format
for the editable data that knows about the data (fieldtype + fieldname +
value), it makes it a lot easier to get this working.



In the past, we've used Macros and they worked fine. They're a bit more
annoying to extend with optional parameters and you have to repeat the
field/property name, but that's not terrible since you can generally get it
down to compile-time errors if you get something wrong (other than
forgetting to add a new field).



-Doug





On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <***@tinco.nl> wrote:

I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.

Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Adrian Stephens]

Yes, you *can* keep your class definitions with the assets, and have external tools read them - but my point is that if you're going to be writing something that parses these definitions, you don't gain much by defining them using contrived macros and templates. I am proposing defining them in a simpler, clearer way that can be easily parsed and spat out as headers, or whatever else you need.

Assets can embed their structure definitions within themselves - they're trivially small binary packets - and since tools contain their own sense of how a 3d scene, say should be stored, the metadata system can modify the data on the fly, rather than require everything to be re-exported because you added a 'flags' field (a rule of game development is that *everything* needs a flags field sooner or later!)

Actually, I just re-read your first paragraph and I think I misunderstood what you were saying. So I'll amend my response to say that unless you store *all* your definitions in *all* your assets, you still won't be able to embed data in a MAX file, for instance. For that sort of thing, we define 'entities' in a (not-C) header file, which is used by our plugin to create a UI for whatever trigger/sound effect/whatever the artist wants to embed.

Ultimately, most exported assets are kept in a platform-neutral state and are turned into platform-specific forms during the build. The platform-specific format definitions don't need to be specified using the metadata script - anything that didn't know about the internals of a PS3, say, wouldn't be able to do anything with a PS3 texture anyway, which I store as a bunch of register values and an offset into a big VRAM block.

Adrian

If you define your reflection in C++, you can still have external tools for asset processing, because class definitions must be stored along with assets. This is something you have to do regardless how you implement your reflection (assuming you don’t have cumbersome “convert all asset to new version whenever the class definition changes”), because different assets may use different versions of class definition and you must be able to read & convert the old data to new version when reading it.
Beyond basic class reflection, the reflection system must also be able to handle special cases. For example when dealing with textures, you reflection system must support “custom reflection” definitions where textures are read straight to the video memory without intermediate data store. Or some classes may expose different set of members depending on the instance of the class (think of exposing variants for example). I’m not sure how you would implement these with reflection defined in script without special hacks in the reflection system.
Cheers, Jarkko
Sent: Monday, March 18, 2013 7:31 PM
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens
That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Jarkko Lempiainen]

If you plan to interpret data, you need to parse those class definitions
anyway, regardless how you define & store them. Furthermore, you *have to*
embed the definitions in asset files if you don't want to do mass asset
conversions every time you change definitions + you can't guarantee access
to all the data using specific definition anyway. These asset files also
need to store only definitions for objects that are stored within that given
asset file, not all your definitions (wasn't sure if this was what you
actually meant, but wanted to clear that up).



So, if you add a new variable to your textures (e.g. color space variable to
define how to interpret the texture), you have to rebuild all your texture
assets? I don't see why you wouldn't just use the same reflection system for
all the data. Another example is container classes, which I have seen
requiring special treatment by reflection systems. I think reflection system
needs to be powerful enough to be usable for all type of data and that you
shouldn't need to add special cases for handing different types anywhere
outside the type definition. Otherwise I end up stabbing the reflection
system every time I have a special case.





Cheers, Jarkko





From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Adrian
Stephens
Sent: Monday, March 18, 2013 10:44 PM
To: sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)



Yes, you *can* keep your class definitions with the assets, and have
external tools read them - but my point is that if you're going to be
writing something that parses these definitions, you don't gain much by
defining them using contrived macros and templates. I am proposing defining
them in a simpler, clearer way that can be easily parsed and spat out as
headers, or whatever else you need.



Assets can embed their structure definitions within themselves - they're
trivially small binary packets - and since tools contain their own sense of
how a 3d scene, say should be stored, the metadata system can modify the
data on the fly, rather than require everything to be re-exported because
you added a 'flags' field (a rule of game development is that *everything*
needs a flags field sooner or later!)



Actually, I just re-read your first paragraph and I think I misunderstood
what you were saying. So I'll amend my response to say that unless you
store *all* your definitions in *all* your assets, you still won't be able
to embed data in a MAX file, for instance. For that sort of thing, we
define 'entities' in a (not-C) header file, which is used by our plugin to
create a UI for whatever trigger/sound effect/whatever the artist wants to
embed.



Ultimately, most exported assets are kept in a platform-neutral state and
are turned into platform-specific forms during the build. The
platform-specific format definitions don't need to be specified using the
metadata script - anything that didn't know about the internals of a PS3,
say, wouldn't be able to do anything with a PS3 texture anyway, which I
store as a bunch of register values and an offset into a big VRAM block.



Adrian



On Mar 18, 2013, at 5:33 PM, Jarkko Lempiainen wrote:





If you define your reflection in C++, you can still have external tools for
asset processing, because class definitions must be stored along with
assets. This is something you have to do regardless how you implement your
reflection (assuming you don't have cumbersome "convert all asset to new
version whenever the class definition changes"), because different assets
may use different versions of class definition and you must be able to read
& convert the old data to new version when reading it.



Beyond basic class reflection, the reflection system must also be able to
handle special cases. For example when dealing with textures, you reflection
system must support "custom reflection" definitions where textures are read
straight to the video memory without intermediate data store. Or some
classes may expose different set of members depending on the instance of the
class (think of exposing variants for example). I'm not sure how you would
implement these with reflection defined in script without special hacks in
the reflection system.





Cheers, Jarkko





From: <mailto:sweng-gamedev-***@lists.midnightryder.com>
sweng-gamedev-***@lists.midnightryder.com [
<mailto:sweng-gamedev-***@lists.midnightryder.com>
mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Adrian
Stephens
Sent: Monday, March 18, 2013 7:31 PM
To: <mailto:sweng-***@midnightryder.com>
sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)





I'm glad we're still talking about this - I think it's important and I
wanted to add my own observations:



The macro/template approach to reflection only applies to one side of the
data/code interface. If all you need to do is serialize runtime data to be
read back in by runtime code, then you're ok; but if you want to access that
metadata from an external tool, you either have to recompile the tool or
plugin when anything changes, or write something that parses that code.



If you're going to have to write a parser anyway, it seems to me you might
as well use it for both code and data views. Which means you can use a more
natural syntax for your structure definitions that even non-programmers can
use.



To map this back into your C++ code, you can pretty trivially write out
header files containing typedefs that correspond to the metadata, and
generate whatever you need to allow the runtime to interpret serialized
data. This also address the separation of code and assets: we don't want
artists having to sync the code, and this keeps the data definitions with
the assets that they define.



This is what I do, anyway. I have a data language, which defines types, and
which can also be used to define data, and all data is stored using these
definitions. This includes bitmaps, sounds, models, etc. The data language
is separated from the data representation so this doesn't imply that bitmaps
are stored in a script-like array of colors - but it does mean that tools
can use the reflection system on more classes of data than just small
gameplay entities. And when distinctions like that are removed, and all
things are treated the same way, lots of clever things just fall out.



Some other random thoughts:

As far as I can see, all C++ needs to allow full reflection is some sort of
fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed
straight into email, so probably bollocks - and not representative of my
actual system!)



struct metadata {

enum TYPE {INT, PTR, ARRAY, STRUCT} type;

metadata(TYPE &_type) : type(_type) {}

};

template<typename A, typename... B> struct md_fields : md<A>, md_fields<B>
{};

template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() :
metadata(STRUCT) {} };

template<typename T> struct md<T*> : metadata { md<T> t; md() :
metadata(PTR) {} };

template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t;
md() : metadata(ARRAY), n(N) {} };

template<> struct md<int> : metadata { md() : metadata(INT) {} };



template<typename T> metadata *get_md(const T &t) { static md<T> m; return
&m; }







Adrian Stephens



On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote:






That sounds pretty similar to the goals of the engine that we're working on
-- basically faster iteration times for everyone (this includes engine
programmers as well).



All of our code (including the compiler) is written in a C#-like language,
so we have perfect reflection from that. Attributes are pretty much
required for providing additional information to property editors for things
like valid ranges of a field or notifications of a property change (most of
which can go away in Final builds).



Another thing to consider may be allowing for data derivation. This can
save quite a bit of work when the need for copies of objects come up that
only need one or two properties changed. If you pick a serialization format
for the editable data that knows about the data (fieldtype + fieldname +
value), it makes it a lot easier to get this working.



In the past, we've used Macros and they worked fine. They're a bit more
annoying to extend with optional parameters and you have to repeat the
field/property name, but that's not terrible since you can generally get it
down to compile-time errors if you get something wrong (other than
forgetting to add a new field).



-Doug





On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <***@tinco.nl> wrote:

I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.

Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list

http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
Sweng-***@lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com



_______________________________________________
Sweng-Gamedev mailing list
Sweng-***@lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com



_______________________________________________
Sweng-Gamedev mailing list
<mailto:Sweng-***@lists.midnightryder.com>
Sweng-***@lists.midnightryder.com

<http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

[Massimo Del Zotto]

As a side note, the Bullet physics API serializes everything with its
accompanying DNA as far as I know.
I'd just throw the link in for documentation.
http://bulletphysics.org/mediawiki-1.5.8/index.php/Serialization_DNA_restrictions,_size_and_alignment_rules

I have no idea if this qualifies as "reflection" but it seems Jarkko'
suggestion is really good habit to me.

Massimo

[Jarkko Lempiainen]

While ago I wrote an introductory article to C++ reflection system I
implemented, which may answer some of your questions about reflection in
general as well:
http://community.spinxengine.com/content/40-introduction--sxp-reflection-sys
tem.html

Simply put a class reflection exposes class members in run-time and you can
build other more complex systems (e.g. networking, editor property grid,
versioned save/load, etc.) on top of it without having to write specific
code for each class for these systems.





Cheers, Jarkko





From: sweng-gamedev-***@lists.midnightryder.com
[mailto:sweng-gamedev-***@lists.midnightryder.com] On Behalf Of Massimo
Del Zotto
Sent: Tuesday, March 19, 2013 3:23 PM
To: sweng-***@midnightryder.com
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)



As a side note, the Bullet physics API serializes everything with its
accompanying DNA as far as I know.

I'd just throw the link in for documentation.

http://bulletphysics.org/mediawiki-1.5.8/index.php/Serialization_DNA_restric
tions,_size_and_alignment_rules



I have no idea if this qualifies as "reflection" but it seems Jarkko'
suggestion is really good habit to me.



Massimo

[Adrian Stephens]

Yes, sorry - the first part of my response was addressing keeping C header files with the assets. When I re-read what you wrote I realized you were talking about keeping the metadata as part of each asset, and I agree with that. My second point was that external tools need to allow people to create new entities based on some list of templates, and for that the tools still need access to those definitions. If all you need to do is 'process' existing assets, the embedded data would be enough.

If I add a new variable to my textures, old textures will still load and be given the default value for that field. I think we agree on why and how that's a good thing!

I do use the same reflection system for all the data - there really are no special cases - but the the language you use to specify a type definition is always going to be turned into some internal representation at some point; how you express those types is not as important as the ability to treat the data in a uniform way in code. So - in the interest of full disclosure - some of my reflection data *is* generated using templates and macros in C++ files! Perhaps that's cheating, but a platform-specific texture, say, is meaningless outside the code that generates it and the code that uses it. There's nothing to be gained by deferring to an external language for that.

Perhaps the 'system' part of 'reflection system' needs more consideration. The important part to me is the ability to treat all data uniformly and automatically, and empower artists and designers to create, browse and combine assets arbitrarily. I don't really care where the types or the data come from as long they can be uniformly interpreted by tools and game code. You shouldn't need to stab a reflection system whose only goal is this. If you want to create some types in code, or javascript or embedded in flash it's no problem as long as the end result is interpretable by everyone else.

Adrian

If you plan to interpret data, you need to parse those class definitions anyway, regardless how you define & store them. Furthermore, you *have to* embed the definitions in asset files if you don’t want to do mass asset conversions every time you change definitions + you can’t guarantee access to all the data using specific definition anyway. These asset files also need to store only definitions for objects that are stored within that given asset file, not all your definitions (wasn’t sure if this was what you actually meant, but wanted to clear that up).
So, if you add a new variable to your textures (e.g. color space variable to define how to interpret the texture), you have to rebuild all your texture assets? I don’t see why you wouldn’t just use the same reflection system for all the data. Another example is container classes, which I have seen requiring special treatment by reflection systems. I think reflection system needs to be powerful enough to be usable for all type of data and that you shouldn’t need to add special cases for handing different types anywhere outside the type definition. Otherwise I end up stabbing the reflection system every time I have a special case.
Cheers, Jarkko
Sent: Monday, March 18, 2013 10:44 PM
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)
Yes, you *can* keep your class definitions with the assets, and have external tools read them - but my point is that if you're going to be writing something that parses these definitions, you don't gain much by defining them using contrived macros and templates. I am proposing defining them in a simpler, clearer way that can be easily parsed and spat out as headers, or whatever else you need.
Assets can embed their structure definitions within themselves - they're trivially small binary packets - and since tools contain their own sense of how a 3d scene, say should be stored, the metadata system can modify the data on the fly, rather than require everything to be re-exported because you added a 'flags' field (a rule of game development is that *everything* needs a flags field sooner or later!)
Actually, I just re-read your first paragraph and I think I misunderstood what you were saying. So I'll amend my response to say that unless you store *all* your definitions in *all* your assets, you still won't be able to embed data in a MAX file, for instance. For that sort of thing, we define 'entities' in a (not-C) header file, which is used by our plugin to create a UI for whatever trigger/sound effect/whatever the artist wants to embed.
Ultimately, most exported assets are kept in a platform-neutral state and are turned into platform-specific forms during the build. The platform-specific format definitions don't need to be specified using the metadata script - anything that didn't know about the internals of a PS3, say, wouldn't be able to do anything with a PS3 texture anyway, which I store as a bunch of register values and an offset into a big VRAM block.
Adrian
If you define your reflection in C++, you can still have external tools for asset processing, because class definitions must be stored along with assets. This is something you have to do regardless how you implement your reflection (assuming you don’t have cumbersome “convert all asset to new version whenever the class definition changes”), because different assets may use different versions of class definition and you must be able to read & convert the old data to new version when reading it.
Beyond basic class reflection, the reflection system must also be able to handle special cases. For example when dealing with textures, you reflection system must support “custom reflection” definitions where textures are read straight to the video memory without intermediate data store. Or some classes may expose different set of members depending on the instance of the class (think of exposing variants for example). I’m not sure how you would implement these with reflection defined in script without special hacks in the reflection system.
Cheers, Jarkko
Sent: Monday, March 18, 2013 7:31 PM
Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++)
The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code.
If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use.
To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define.
This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out.
As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!)
struct metadata {
enum TYPE {INT, PTR, ARRAY, STRUCT} type;
metadata(TYPE &_type) : type(_type) {}
};
template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {};
template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} };
template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} };
template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };
template<> struct md<int> : metadata { md() : metadata(INT) {} };
template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; }
Adrian Stephens
That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well).
All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds).
Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working.
In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field).
-Doug
I think the point of the discussion is to not just assume C++ isn't
going to do it. As far as I understand the goal is to have the ability
to develop tools that aid programmers and artist in iterating rapidly.
This means for artists that they can swap artwork in at runtime or
quickly boot the game from a serialized state with new artwork, not
just on his/her own computer but also over a network to perhaps a
console. For programmers this may mean that they can view the game
state easily, launch the game in various states easily, swap in/out
scripts easily, maybe even change code without fully recompiling or
even restarting the game.
Am I missing something important?

Post by Massimo Del Zotto
I have limited experience but I have been very impressed by the separation
provided by scripting languages.
My goal is to have fast iteration times, both artists and programmers
It appears to me C++ isn't going to do it. Especially on the artist side of
the thing.
By the way, I would like to have an insight on how this reflection
information is to be used. If it is used to simulate duck typing I'd use a
tool which supports duck typing natively.
Massimo
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com

_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
_______________________________________________
Sweng-Gamedev mailing list
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com